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methods  vithin  individually  Prescribed  Instruction*  Appendix  XI  deals 
vith  possible  further  developments  in  decision*theory  application* 
Appendix  III  treats  the  coi^^^mathematical  theory  underlying  the 
proposed  applications^  and  Appendix  IV  deals  vith  the  computer 
problems  involved  in  such  applications*  (RC) 
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testing  time  low  and  specifically  to  keep  test  lengths  below  the  20-item 
mark,  and  preferably  within  the  6  to  12-item  range*    If  this  desire  is 
to  he  satisfied,  and  at  the  same  time  we  are  to  assure  ourselves  that 
accurate  decisions  are  being  made,  on  the  average,  then  most  of  the 
time  there  will  be  a  need  to  bring  some  further  information  Into  the 
decision-^making  process* 

There  is  no  difficulty  in  recognizing  that  such  information  exists, 
in  the  background,  and  is  available  for  use*    The  difficulty  is  one 
of  finding  the  technology  for  quantifying  this  Information  and 
incorporating  it  into  the  dec is ion -making  process*    One  of  the  things 
that  we  know  about  individualized  instructional  programs,  for  example 
the  University  of  Pittsburgh's  Individxially  Prescribed  Instructional 
(IPI)  Program,  is  that  students  take  very  short  curriculum  embedded 
tests  within  each  module,  and  continue  to  receive  Instruction  until 
Such  time  as  there  is  good  reason  to  believe  that  they  can  pass  the  end 
of  module  posttest*    Thus  the  very  nature  of  the  IPI  module  is  such 
as  to  reduce  the  variability  of  posttest  performance  levels,  and  to 
suggest  that  there  is  reasonable  prior  probability  that  any  particular 
student  will  pass  the  posttest*    It  is  precisely  this  information 
which  can  and  needs  to  be  incorporated  in  the  decision  analysis  which 
has  a  short  criterion**ref erenced  posttest  as  its  direct  data  input. 

These  considerations  clearly  suggest  that  Bayesian  methods  which 
combine  prior  (that  is,  background)  information  with  direct  observational 
inforrnation,  may  be  useful  in  sharpening  IPI  decision  making*    We  shall 
therefore  turn  to  the  technical  problems  of  qxiantifying  this  background 
information  for  a  Bayesian  analysis.    If  IPI  proErams  were  uniformly 
administered  throughout  the  country,  it  would  be  possible  to  gather 
background  information  at  various  locations  and  thus  to  construct 
data-abased  prior  distributions  for  adoption  at  all  IPI  Installations* 
Unfortunately,  from  the  decision  making  point  of  view,  but  not 
necessarily  from  the  educational  point  of  view,  IPI  methods  are 
administered  with  substantial  local  flexibility*    As  a  result,  data 
gathered  from  one  Installation  may  not  necessarily  be  relevant  to  the 
implementation  of  IPI  at  another  installation* 

At  a  particular  IPI  installation  at  which  the  instructional 
process  has  stabilized  and  made  uniform  and  where  historical  records 
are  available,  it  will  be  possible  to  use  standard  Bayesian  methods 
to  develop  an  :Lnstructional  decision  making  process.    However,  as  a 
fast  growing  and  evolving  entity,  IPI  is  finding  itself  being 
modified  continuously  in  current  schools  and  being  introduced  into 
new  schools  where  such  background  information  has  not  been  gathered, 
and  therefore  standard  Bayesian  techniques  cannot  be  used  with  any 
expectation  of  guaranteeing  high  decision-making  accuracy.  It 
therefore  becomes  necessary  to  devise  and  Implem^t  new  Bayesian 
methods  that  make  it  possible  to  simultaneously  gather  the  background 
Information  necessary  to  provide  prior  information  and  at  the  s,^me 
time,  gather  and  integrate  the  direct  information  concerning  the 
performance  on  each  individual  student ,    It  is  this  requirement  for 
simultaneous  data  gathering  that  demanded  the  development  of  new  and 


complex  statistical  methods^  that  make  It  possible  to  simultaneously 
incorporate  both  direct  observations  and  collateral  group  Information* 

A  gross  statistical  methodology  for  accoinplishing  this  has  been 
available  in  embryonic  form  since  the  early  days  of  Truman  Kelley's 
Statistical  Methods  text  (1923)*    Kelley  concerned  himself  with  the 
estimation  of  true  score  for  a  group  of  individuals  and  showed*  by  a 
standard  application  of  regression  theory*  that  improved  estimates  of 
true  score  could  be  obtained  through  a  simultaneous  estimation 
procedure*  based  both  on  the  direct  observations  of  the  particular 
individual  and  estimates  of  the  group  mean  and  true  score  variation 
across  individuals*  all  made  simultaneously*    The  celebrated  Kelley 
formula  is  of  a  form  which  estimates  each  person^s  true  score  as  a 
weighted  average       his  observed  score*  and  the  mean  of  the  observed 
scores  throughout  the  population*    Thus*  each  individual's  true  score 
estimate  is  regressed  from  his  observed  score,  towards  the  overall  mean 
observed  score  In  the  population*    A  specific  formula  was  given  by 
Kelley  which  showed  the  extent  to  which  improvement  1.1  estimation 
could  be  accomplished  using  this  technique*    In  1956*  Charles  Stein 
used  3i  similar  kind  of  logic  to  show  that  in  most  situations*  when  a 
group  of  individual  parameters  are  being  estimated  simultaneously*  the 
standard  estimates*  which  here  correspond  to  the  mean  scores  for 
each  of  the  Individuals*  are  inadmissible  in  a  strict  statistical 
sense  in  that  there  are  always  available  estimates  with  better  mean- 
squared  error*    Stein  proposed  a  class  of  estimates  of  a  form 
remarkably  similar  to  those  given  by  Kelley*    in  fact  they  differ  from 
Kelley's  estimates  only  to  the  extent  that  the  regression  to  the  mean 
is  statistically  somewhat  less  than  with  the  Kelley  estimates*  Stein's 
work  opened  up  the  whole  field  of  simultaneous  estimation*  and  his 
ideas  stimulated  similar  developments  from  other  classical*  from 
Bayesian*  and  from  empirical  Bayesian  points  of  view*    From  a  Bayesian 
point  of  view  the  first  comment  on  this  possibility  was  made  by 
Lindley  (1962)  in  the  discussion  at  the  Royal  Statistical  Society  of  a 
second  paper  by  Stein  (1962)*    A  paper  by  Box  and  Tiao  (1968)  on  the 
Bayesian  Estimation  of  Means  for  the  Random  Effects  Model*  provided 
a  specific  Bayesian  methodology*  though  Box  and  Tiao  did  not  directly 
confront  the  simultaneous  estimation  problem*    Recently  this  approach 
has  been  taken  up  in  great  detail  by  Lindley  and  his  associates*  and 
a  general  formulation  has  been  provided  in  a  paper  by  Lindley  and 
Smith  (1972)*    The  possibility  of  application  of  these  methods  in 
educational  research  was  noted  several  years  earlier  by  Novick  (1970)* 
Building  upon  the  work  of  Lindley  (1971)*  Box  and  Tiao  (1968)*  Stein 
(1956*  1962)*  Kelley  (1923),  and  others*  Novick*  Lewis*  and  Jackson 
(1973)  specialized  the  Bayesian  simultaneous  estimation  procedure  to 
the  problem  of  the  estimation  of  proportions  in  m-units  and  shortly 
after  doing  this*  as  indicated  in  that  paper*  it  became  clear  that 
this  method  might  have  useful  application  in  the  field  of  criterion- 
referenced  tests*    Stimulated  ty  discussions  with  Ronald  Hambleton* 
who  was  an  ACT  postdoctoral  fellow  during  the  Summer  of  1971*  a 
proposal  was  forwarded  to  the  Office  of  Education  for  the  funding  of 
a  project  to  further  develop  and  tailor  these  Bayesian  methods  for 


potential  Implementation  within  IPX*    In  thi?  proposal  it  was  pointed 
out  that,  in  theory,  these  methods  could  increase  the  ef f ective^test 
length  the  equivalent  of  6  to  25  observations  through  the  use  of 
collateral  information*    To  put  it  the  other  way  around,  this  allows 
a  reduction  In  the  required  test  length  anywhere  from  6  to  25  items 
while  maintaining  the  same  level  of  precision* 

It  was  also  noted  in  the  proposal  that  the  primary  new  statistical 
development  would  focus  on  a  shift  In  strategy  from  the  point  estimation 
of  individual  ability  levels,  or  true  performance  levels,  to  the 
determination  of  the  probability  that  a  particular  individual  mastery 
level  is  larger  than  some  specified  criterion  level*    From  an 
educational  point  of  view,  this  represented  a  tailoring  of  the  theoiy 
to  criterion^ref erenced  testing  rather  than  norm-ref erenced  testing* 
Statistically,  this  meant  that  the  output  of  the  Bayesian  analysis  would 
not  be  a  joint-point  (modal)  estimate  of  ability  scores  for  students^ 
but  rather  for  each  student,  the  aposterlori  detennination  of  the 
probability  that  his  mastery  score  Is  larger  than  some  specified^ 
criterion  level*    The  strategy  as  noted  in  the  proposal  would  be  to  use 
these  aposterlori  probabilities  in  a  standard  Sayesian  decision 
theoretic  context  for  deciding  whether  or  not  an  Individual  student 
should  be  advanced  to  the  new  unit  of  instruction  or  retained  for 
further  work  in  the  current  module*    In  the  proposal  it  was  suggested 
that  the  simplest  reasonable  approximation  to  reality  would  be  to 
assume  a. threshold  loss  function  which  specified  zavo  losses  for 
correct  positive  and  negative  decisions,  and  losses  ja  and  b^, 
respectively,  (a,  b  >  0),  for  false  positive  and  false  negative 
decisions*    Por  the  most  part  this  loss  structure  is  assumed  throughout 
the  work  on  this  project  though  in  one  paper  we  do  indicate  that  this 
is  only  a  first  approximation  to  reality,  and  that  other  more 
reasonable  appro:::imations  should  be  considered  In  future  work*  We 
do  expect  that  procedures  developed  here,  based  on  threshold  loss» 
will  be  a  very  good  and  workable  first  approximations  indeed* 

In  developing  our  materials  for  this  project  we  have  been 
cognizant  of  the  fact  that  several  different  kinds  of  technical 
questions  would  need  to  be  resolved,  and  that  several  different  kinds 
of  audiences  would  need  to  be  addressed  in  our  final  report*    This  has 
led  us  to  seek  collaboration  with  persons  more  experienced  in  IPI 
methods  and  the  preparation  of  several  somewhat  overlapping  expository 
papers,  in  addition  to  our  technical  papers*    First  let  us  consider  the 
technical  questions*    The  primary  problem  was  to  work  with  the  Bayesian 
simultaneous  decision  model  for  the  estimation  of  proportions  in 
m'-units,  and  to  derive  from  the  joint  posterior  distribution  on  these 
parameters,  the  marginal  distributions  for  each  individual  element* 
Beyond  this  there  was  a  further  desire  to  use  this  posterior 
distribution  in  a  full  dec is ion* theoretic  analysis*    Furthermore,  there 
was  a  desire  to  utilize  in  the  analysis,  not  only  a  general  level  of 
collateral  information  concerning  the  general  level  of  performance  of 


4 


ERIC 


8 


other  students  In  the  Instructional  unlt»  but  also  the  performance  of 
the  particular  student  on  other  instructional  units*    This  would  be 
particularly  useful  in  placement  testing^  where  a  student  takes  short 
tests  on  several  possibly  highly  related  instructional  objectives*  The 
technical  results  required  for  these  analyses  are  contained  in 
papers  by  Hovick,  L  ewis^  and  Jacicson  (1?73^  Appendix  3*1)^  L^wis* 
Wang,  and  Novick  (1973,  Appendix  3*2),  Lewis,  Wang,  and  Novick  (1973, 
Appendix  3*3),  Wang  (1973,  Appendix  3*4),  Wang  and  Lewis  0.973, 
Appendix  3*5),  and  Wang  and  Lewis  (1973,  Appendix  3*6),  as  listed  in 
our  overview  summary  that  follows  shortly*    These  papers  tend  to  be 
rather  technical  in  nature,  and  if  one  wishes  to  read  more  than  the 
introductory  and  summary  siiatcments,  one  will  need  to  follow  some 
detailed  statistical  and  mathematical  arguments*    Nonetheless,  it  was 
necessary  to  present  this  material  in  a  rigorous  technical  form  so 
that  researchers  wishing  to  extend  these  results  would  have  a  basis  for 
such  extension*    These  papers  can  be  found  in  Appendix  Number  3  to 
this  report* 

The  general  question  of  the  application  of  statistical-decision 
theory  would,  in  an  IPI  context,  be  one  which  we  felt  required  fairly 
extensive  discussion,    While  the  threshold^lo&s  function  that  we  have 
adopted  is  certainly  a  useful  one,  ard  indeed  we  mean  to  have  the 
results  derived  from  it  taken  seriously,  we  do  believe  further 
significant  improvements  Liay  be  possible,  using  more  sophisticated  lose 
functions*    To  make  this  further  work  possible,  we  have  included  a 
rather  lengthy  primer  on  decision  analysis  for  Individually  Prescribed 
Instruction  which  comprises  the  content  of  Appendix  Number  2*    It  is 
our  hope  that  persons  within  IPI>  and  those  associated  with  other 
individualized  inst Aictional  programs  will  give  this  paper  some  study, 
and  hopefully  a  dialogue  vi')!  ensue  among  such  people  discussing  the 
relative  merits  and  demerits  of  various  possible  loss  functions* 

We  have  also,  in  this  project,  been  cognizant  of  the  fact  that 
the  procedures  we  are  proposing  are  extraordinarily  complicated,  both 
theoretically  and  practically*    Yet  we  intend  that  these  procedures 
bj  adopted  for  classroom  use  by  persons  whose  professional  skill  lie 
in  instruction  and  not  in  theoretical-educational  measurement  or 
statistical  decision  theory.    Thus  we  knew  we  would  need  to  provide 
means  for  making  these  procedures  available  in  a  simple  format  for 
classroom  use*    Vfe  have  attempted  to  accomplish  this  in  two  ways* 
First  of  all,  in  our  theoretical  appendix  (Appendix  Number  3),  we 
have  provided  a  set  of  tables  (Wang,  1973,  Appendix  3*4)  which  indeed 
drastically  simplify  the  computational  work  in  Bayesian  IPI  decision 
making*    We  might  also  refer  to  a  paper  in  development  by  Millman 
(in  preparation)  which  gives  a  detailed  numerical  example  applying 
our  methods  and  these  tables  to  an  IPI  decision-making  problem* 

However,  we  did  not  feel  that  this  approach  would  be  entirely 
satisfactory*    Our  feeling  has  been,  and  remains,  that  the  whole 
arithmetic  process  required  for  decision  making  will  best  be  done,  in 
toto,  by  a  computer*    We  note  that  mini-computers  are  now  in  use 


within  IFI»  and  with  the  continuing  reduction  In  costs  of  such 
equipment »  we  can  speculate  that  In  the  future  IPX  wlll»  In  Its 
£3tandard  form^  be  monitored  In  a  computers-based  environment »  The 
question  then  was^  could  :  e  take  our  decision  making  procedures  and 
computerize  them  In  such  a  way  that  this  enormously  complicated  and 
sophisticated  machinery  could  be  used  by  persons  having  a  verbal- 
theoretical  understanding  of  what  was  being  done»  but  little  precise 
understanding  of  the  sophisticated  mathematical  and  statistical 
theory  underlying  the  given  formulas* 

Fortuitously,  the  principal  investigator  has  been  involved, 
concurrently,  in  a  project  concerned  with  the  interactive  conversational 
analysis  of  data  using  Bayesian  methods*    It  therefore  seemed 
appropriate  that  some  additional  efforts  be  made  in  this  area,  and 
thst  the  result  from  this  additional  effort  be  reported  as  part  of 
this  project*    The  problems  faced  in  attempting  to  provide 
conversational  language  programs  to  monitor  IPI  are  identical  with 
those  in  other  conversational  statistical  applications*    These  are 
discussed  in  detail  by  Novick  (1973,  Appandix  4*1),  Isaacs  (1972, 
Appendix  4.2),  Isaacs  (1973,  Appendix  4*3),  and  Christ  (1973, 
Appendix  4*4)* 

In  summa.ry  then.  Appendix  Number  2  deals  witti  possible  further 
developments  In  dec  is  ion*"  theory  application.  Appendix  Nximbcr  3  deals 
with  the  core-mathematical  theory  underlying  our  proposed  applications, 
and  Appendix  Number  4  deals  with  the  computer  problems  involved  in 
such  applications*    For  most  readers  however,  the  papers  of  greatest 
interest  will  be  those  contained  in  Appendix  Number  1.    Here  four 
papers  are  given  which  are  concerned  directly  with  the  Implementation 
of  these  new  methods  within  Individually  Prescribed  Instruction* 
These  papers  should  probably  be  read  in  the  order  in  whjch  they  appear* 

The  paper  by  Novick  and  Lewis  on  Prescribing  Test  Length  for 
Criterion-^Referenced  Measurement  (1973,  Appendix  1*1),  in  fact  is  much 
more  general  than  its  title  would  Indicate*    It  is  rather  a  careful 
laying  out  and  consideration  of  all  the  factors  which  must  be  taken 
into  account,  both  in  the  actual  decision  process  and  in  the 
consideration  of  necessary  length  for  criterion-'ref erenced  tests*  One 
of  the  difficulties,  we  think,  in  attempting  to  apply  decision  theory 
In  IPI,  is  that  some  of  these  considerations  have  not  been  discussed 
In  the  literature,  and  therefore  there  is  insufficient  guidance  on 
these  matters*    Specifically  isolated  for  consideration  are:    1)  the 
current  level  of  functioning  of  the  student,  2)  the  minimum  acceptable 
criterion  level  to  certify  mastery,  3)  the  prior  probability  that  a 
student  has  attained  mastery,  4)  the  loss  ratio  for  false  positive 
and  false  negative  decisions,  and  5)  the  premium  on  testing  time 
within  the  instructional  process*    Following  a  discussion  of  these 
topics  it  becomes  possible  to  intelligently  investigate  the  test  length 
problem  and  to  give  some  tentative  recommendations.    Hambleton  and 
Novick  (1973,  Appendix  1,2),  explicate  some  of  the  ideas  contained 
in  the  paper  of  Ncvick,  Lewis,  and  Jackson  (1973,  Appendix  3<1), 
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In  a  less  theoretical  language  and  provide  a  brief  Introduction  to 
the  threshold  loss  paradigm^    Ferguson  and  Novlck  (1973,  Appendix  1.3), 
give  some  details  on  precisely  how  these  methods  can  be  implemented 
within  Indlvldxtally  Prescribed  Instruction,  and  Hambleton  (1973, 
Appendix  1.4),  broadens  the  perspective  by  indicating  the  relevance 
of  these  methods  both  within  IPX  and  for  other  instructional  programs 
such  as  Project  Plan, 


II*  OVERVIEW 


In  Jactuaiy  of  1972»  the  United  States  Department  of  Healthy  Education^ 
and  Welfare  awarded  a  grant  of  $99,492  to  Dr*  Melvin  Noviclc,  Director 
ot  ACT^s  Psychonetric  Research  Depi^rtment  for  a  study  on  "New 
Statistical  Techniq.ies  to  Evaluate  Criterion-Referenced  Tests  Used 
in  Individually  Prescribed  Instruction."    The  focus  of  the  project 
has  been  on  che  application  of  certain  Bayesian  methods  introduced 
by  Professor  D*  V.  Lindley  of  the  University  College  London  whose 
research  b^s  been  supported  in  part  by  ACT  for  the  past  three  years* 
Vforlc  by  Melvin  R»  Noviclc,  Charles  Lewis,  and  Paul  lU  Jackson,  on  the 
Bayesian  estimation  of  proportions  in  tn  gtoups,  released  as  ACT 
Technical  Bulletin  No*  1,  and  subsequently  published  in  PsychometrilCa^ 
has  been  the  initial  take-off  point  for  applications  in  this  project* 
This  work  suggests  that  in  a  criterion^referenced  measurement 
situation,  an  increase  in  precision  equivalent  to  adding  from  six 
to  twenty^five  item^  can  be  attained  by  using  the  Bayesian  method* 
In  this  newly  defined  approach,  the  estimation  of  mastery  scores  is 
replaced  by  the  determination  of  the  probability  that  the  true 
mastery  scores  are  larger  than  some  specified  criterion  level*  The 
result  of  this  research  is  the  creation  of  a  new  test  theory  for 
Individually  Pr^^scribed  Instruction,  and  a  statistical  and  compu- 
tational technology  for  implementing  this  theory* 

A  bibliography  of  the  papers  completed  for  this  project,  with 
annotations  indicating  how  each  pap^  fits  into  the  oVarall  project 
development  follows. 


Annotated  Bibliography 

1*1    Novick,  M*  R*,  &  Lewis,  C*    Prescribing  test  length  for 

criterion^referenced  meaBurement*    ACT  Technical  Bulletin 
No*  18*    Iowa  City,  loVai    The  American  College  Testing 
Program,  1973* 

This  bulletin  demonstrates  the  effectiveness  of  the  eight  to 
twelve  item  criterion-^referenced  tests  in  placement  pre  and 
posttesting  tdien  the  new  Bayesian  methods  of  analysis  are  used* 
It  is  also  noted  here  that  Bayesian  techniques  can  be  applied 
sequentially  without  modification,  and  thus  all  of  the  benefits 
of  sequential  analysis  are  available  without  further  complicating 
the  analysis*    In  |;he  introduction,  work  ot  Millman  is  used  to 
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demonstrate  the  Inadequacy  of  classical  analysis.    Specific  test 
length  reconmendatlons  are  given  dependent  upon  (1)  the  loss 
ratlo»  (2)  prior  probabilities »  and  (3)  the  specified  criterion 
level. 

1.2  Hambleton,  R.  K. ,  &  Novlck,  M.  R.    Toward  an  Integration  of 

theory  and  method  for  criterion-referenced  tests. 
Journal  of  Educational  Measurement.  1973,  10(3) >  159-170. 

This  article  d/iscrlbes.  In  nontechnical  language,  the  Ideas  and 
methods  Introduced  In  Reference  3.1,  and  elaborated  In  Reference 
3.2,  to  take  account  of  the  collateral  Information  on  (m-1) 
students  to  help  estimate  the  probability  that  the  mastery 
level  of  each  iL^'th  student  Is  greater  than  the  required 
criterion  level.    The  central  concept  taken  from  References  3.1 
and  3.2,  and  exposlted  here.  Is  thai:  In  Individually  Prescribed 
Instruction  decisions  must  be  based  on  the  aposterlorl  probability 
that  the  student's  level  of  functioning  Is  greater  than  the 
prescribed  criterion  level.    This  approach  is  Illustrated  using 
a  simple  threshold  loss  function  and  a  posterior  marginal 
distribution  that  depends  on  sample,  prior,  and  collateral 
Information. 

1.3  Ferguson,  R.  L. ,  &  Novick,  M.  R.    Implementation  of  a  Bayeslan 

system  for  decision  analysis  in  a  program  of  Individually 
Iirescribed  Instruction.    ACT  Research  Report  No.  60. 
Iowa  City,  Iowa;    The  American  College  Testing  Program, 
1973. 

This  report  provider  some  precise  detail  of  how  the  new  methods 
can  be  used  in  placement  testing,  pretesting,  and  posttestlng. 
The  various  decision  ntodes  of  IPI  are  Identified  and  the  precise 
way  In  which  the  new  techniques  can  be  Implemented  at  each  need 
are  discussed  in  detail. 

1.4  Hambleton,  R.  K.    A  review  of  testing  and  decision-making 

procedures  for  selected  Individualized  Instructional 
programs.    ACT  Technical  Bulletin  No.  15.    Iowa  City, 
Iowa;    The  American  College  Testing  Program,  1973. 

This  bulletin  discusses  the  similarities  and  differences  among 
several  approaches  to  Individualized  Instruction  and  concludes 
that  the  new  Bayeslan  methods  will  be  useful  In  each  of  these 
approaches.    Included  In  the  survey  are  svstems  used  In 
Individually  Prescribed  Instruction,  Project  Plan,  Mastery 
Learning,  and  approaches  to  computer  assisted  Instruction. 
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2,1    Davis,  C,  E,,  Hickman,  J,,  &  Novick,  M.         A  primer  on  decision 
analysis  for  Individually  Prescribed  Instruction,  ACT 
Technical liullvjtin  Mo,  17,    Iowa  City,  Iowa:    The  American 
College  Testing  Program,  1973. 

This  bulletin  provides  an  overview  of  how  the  utilities  of  various 
outcomes  can  be  logically  combined  with  the  aposteriori 
probabilities  of  these  outcomes  to  provide  a  coherent  basis  for 
decision  making.    This  paper  illustrates  in  detail,  results  for 
several  important  utiltiy  functions,  thus  going  beyond  the 
simple  threshold  loss  situation.    For  each  of  these  loss 
functions,  an  illustration  is  given  of  how  to  determine  the 
advance-retain  observed  cut-score, 

3.1  Novick,  M,  R,,  Lewis,  C, ,  &  Jackson,  P,  H,    The  estimation  of 

proportions  in  m  groups,    Psychometrika,  1973,  38,  19-46, 

This  is  the  fundamental  theoretical  paper  which  provides  the 
basis  in  Bayesian  methodology  for  all  of  the  methods,  theory 
and  applications  discussed  in  the  remaining  project  papers.  This 
paper  was  produced  prior  to  the  commencement  of  the  project  and 
was  the  basis  for  the  project  proposal, 

3.2  Lewis,  C, ,  Wang,  M, ,  &  Novick,  M,  R,    Marginal  distributions 

for  the  estimation  of  proportions  in  m  groups,  ACT 
Technical  Bulletin  Ho,  13,    Iowa  City,  Iowa:    The  American 
College  Testing  Program,  1973, 

This  paper  provides  the  key  methodological  development  of  the 
project  "  a  procedure  for  obtaining  the  aposteriori  probability 
of  mastery  for  each  student  individually  from  the  m-group 
proportion  method,  using  a  veil  established  computational 
approach  to  marginalization  due  to  Box  and  Tiao, 

3.3  Lewis,  C, ,  Wang,      ,  &  Novick,  M,  R,    A  proper  prior  for  p^, 

in  estimating  proportions  in  m  groupt,    ACT  Technical 
Bulletin  Supplement  No,  13-1,    Iowa  City,  Iowa:  The 
American  College  Testing  Program,  1973, 

This  supplement  introduces  an  improvement  to  the  basic  theory 
of  Reference  3.2  that  makes  it  possible  to  incorporate  prior 
information  on  the  average  of  the  group  means* 

3.4  Wang,        Tables  of  constants  for  the  posterior  marginal  estimates 

of  proportions  in  m  groups,    ACT  Technical  Bulletin  No,  14, 
Iowa  City,  Iowa;    The  American  College  Testing  Program,  1973, 

These  tables  make  it  possible  to  monitor  Individually  Prescribed 
Instruction  without  dependence  on  a  computer. 
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3.5  Wang,  M, ,  &  Lewis,  C,    Estimation  of  proportions  in  a  two-way 

teble,    ACT  Technical  Bulletin  Mo,  16,    Iowa  City,  Iowa: 
The  American  College  Testing  Program,  1973, 

This  work  makes  it  possible  to  take  Account  of  the  collateral 
information  contained  in  the  test  scores  on  t-1  objectives  as 
well  as  m-1  students  In  estimating  the  proficiency  of  each 
m-th  student  on  each  t-th  objective.    Following  transformation, 
a  full  two-way  analysis  of  variance  is  used,  though  it  is  also 
shofm  In  the  following  paper  that  a  two-way  no  Interaction 
model  consistently  provides  almost  identical  results, 

3.6  Wangj.  M,,  &  Lewis,  c.    Marginal  distribution  for  the  estimation 

of  proportions  in  a  two-way  table,    ACT  Technical  Bulletin 
No,  19,    Iowa  City,  Iowa:    The  American  College  Testing 
Program,  1973, 

An  extension  of  the  previous  paper,  this  bulletin  provides  a 
method  of  assessing  the  probability  of  a  student's  mastery  of 
that  objective,    A  no^uteraction  model  is  used  here  because 
of  computational  difficulties  encountered  in  attempting 
marginalization  with  the  interaction  model, 

4.1  tfovick,  M,  R,    High  school  attainment:    An  example  of  a 

computer-*assisted  Bayesian  approach  to  data  analysis. 
International  Statistical  Review,  1973,  41,  264-271, 

Prepared  prior  to  the  beginning  of  this  project,  this  paper 
demonstrates  how  a  non statistician  can  use  complex  statistical 
techniques  with  the  step*by-step  conversational  guidance  of 
a  system  of  Computer  Assisted  Data  Analysis  (CADA),    As  a 
result  of  this  finding,  we  believe  that  CADA  can  make  it 
possible  for  the  classroom  teacher  to  use  the  sophisticated 
statistical  procedures  developed  in  this  project, 

4.2  Isaacs,  G,  L,    Interdialect  translatability  of  the  BASIC 

programming  language,    ACT  Technical  Bulletin  Mo,  11, 
Iowa  City,  Iowa:    The  American  College  Testing  Program, 
1972, 

A  study  of  the  BASIC  programning  language  showing  how  it 
is  possible  to  program  in  one  dialecc  in  such  a  way  as 
to  facilitate  translation  into  other  dialects,  and  thus 
make  it  possible  to  transport  CADA  programs  to  many  different 
kinds  of  computer  installations,    This  research  makes  it 
possible  to  implement  CADA  mode  IPI  programs  on  any 
adequate  computer  system. 


15 


ERIC 


11 


4.3  Isaacs^  G.  L.    A  tabular  survey  of  basic  computer  systems. 

ACT  Technical  Bulletin  Supplement  No.  11-1.    Iowa  City, 
Iowa:    The  Ainerlcan  College  Testing  Prograio^  1973. 

These  tables  provide  a  feature  by  feature  comparison  of  BASIC 
language  dialects  as  Implemented  on  various  computer  systems 
with  an  evaluation  of  the  adequacy  of  each  dialect  for  CADA 
Implementation.    Particular  emphasis  Is  placed  on  chaining, 
string  handling,  and  formatted  output  capability,  as  these 
are  the  BASIC  features  most  needed  In  CADA.    This  report, 
completed  In  March,  1973,  shows  that  a  large  number  of  computer 
systems  are  adequate  for  CADA  applications.    An  on-going  survey 
of  BASIC  systems  Indicates  that  many  of  these  are  being 
substantially  improved.    Most  of  these  system  updates  should 
be  completed  within  the  next  ninety  days,  shortly  after  which 
a  revision  of  Technical  Bulletin  Supplement  No.  11-1  will  be 
prepared. 

4.4  Christy  D.  E.    The  CADA  monitor.    ACT  Technical  Bulletin  No.  12. 

Iowa  City,  Iowa:    The  American  College  Testing  Program, 
1973. 

This  is  a  description  of  the  Monitor  used  to  organize  the  CADA 
package  of  programs.    The  interrelationship  of  the  programs 
currently  available  on  the  Monitor  is  shown.    Also  the 
design  philosophy,  which  enables  the  programs  to  be  easily 
Interconnected  and  used  by  unsophisticated  investigators,  is 
discussed.    M^ch  of  the  design  philosophy  is  applicable  to  many 
other  interactive  situations,  since  its  main  thrust  is  the 
Improvement  of  the  man-machine  interface^  while  minimizing 
programming  effort. 
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Ill,     SUMMARY  OF  RESULTS 


A.    The  Structure  of  the  Statistical  ^k>nltorlng  System  and  Its 
Implications, 

The  prerequisite  for  the  Introduction  of  a  statistical  monitor  for 
IPI  Is  a  clear  statement  of  the  problem  and  an  understanding  of  the 
evaluations  that  would  need  to  be  made  for  Input  Into  the  decision- 
making process.    In  the  paper  entitled,  '^Prescribing  Test  Length  for 
Criterion-Referenced  Measurement*'  by  Novlck  and  Lewis  (1973,  Appendix 
1»1),  each  of  the  kinds  of  information  required  for  prescribing  test 
length  and  making  decisions  based  on  crlterion*-ref erenced  tests  is 
discussed*    The  five  major  considerations  in  structuring  IPI  decisions 
are: 

(1)  The  current  level  of  functioning  (tt)  of  the  student, 

(2)  The  minimum  advancement  score  (tfq)  required  for  defining 
mastery  of  a  module, 

(3)  Background  infcrtnation  available  on  each  student  and  on  the 
instructional  process, 

(^)    Relative  losses  Incurred  in  making  false  positive  and  false 

negative  decisions,  and 
(5)    llie  premium  on  testing  time  within  the  instructional  process. 

In  criterion-referenced  testing,  we  think  of  a  hypothetical 
(infinitely  large)  pool  of  test  items  relevant  to  a  single  behavioral 
objective.    A  student  is  considered  a  master  of  a  behavioral  objective 
if  the  percentage  of  items  he  would  get  correct  over  the  entire  pool, 
his  level  of  functioning;  (tt)  ,  exceeds  a  specified  criterion  level  (tt^)* 
Because  a  test  contains  only  a  small  sample  from  that  pool,  errors 
in  decision  making  must  be  expected* 

As  a  first  approximation  it  may  be  assumed  that  a  loss    a  is 
Incurred  if  a  student  is  deemed  a  master  when  he  is  not  (a  false 
positive)  loss    b    is  incurred  if  he  is  deemed  a  nonmaster  when  he  is 
not  (a  false  negative),  and  zero  loss  if  a  correct  decision  is  made. 
A  coherent  system  of  decision  making  is  based  on  the  aposterlorl 
probability  that  the  student^s  level  of  functioning  is  above  the 
specified  criterion  level  and  the  ratio  a/b  of  losses  associated  with 
false  positive  and  false  negative  decisions.    The  rule  is  that  a 
student  is  advanced  if  the  ratio  of  the  probability  that  he  is  a 
master  to  the  probability  that  he  is  not,  exceeds  the  ratio  of  false 
positive  to  false  negative  losses.    This  is  equivalent  to  choosing 
that  action  (advance  or  retain)  which  has  the  highest  expected  utility* 

Given  a  prior  distribution  for  a  student* s  level  of  functioning  and 
a  given  test  length  (sample  size)  it  is  possible  to  determine  a 
minimum  advancement  (test)  score  required  to  indicate  proficiency. 
This  is  the  lowest  score  that  will  yield  an  aposterlorl  probability  of 
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mastery  large  enough  to  justify  an  advancement  decision.    The  details 
of  this  analysis  are  given  in  this  paper  (Novick  and  Lewis,  1973; 
Appendix  1.1).    Consideration  is  given  to  specified  criterion  levels 
of  .70,  .75,  .80,  and  .85.    The  loss  ratios  are  assumed  to  take 
the  values  1.5,  2.0,  2.5,  or  3.0.    Thus  it  is  assumed  that  typically 
a  loss  incurred  for  a  false  positive  decision  will  be  at  least  one  and 
one-half  times  that  for  a  false  negative  decision*    Various  prior 
distributions  are  considered  for  each  specified  criterion  level* 
Generally  it  is  assuioed  that  the  prior  distribution  will  have  a  mean 
value  near  the  specified  criterion  level  and  typically  and  desirably 
slightly  larger.    For  each  analysis  with  a  particular  expected  value 
for  the  prior  distribution,  four  different  priors  are  considered  with 
varying  degrees  of  certainty  in  the  prior  distribution.    The  results 
of  these  analyses  are  summarized  in  a  set  of  tables,  seven  through 
eleven  (Appendix  1.1,  Pages  20,  23,  26  and  28).    For  each  combination 
of  specified  criterion  level,  loss  ratio,  and  prior  distribution,  a 
recommended  test  length  and  minimum  advancement  score  are  given.  At 
the  end  of  each  table,  some  general  recommendations  ^re  given  which  seem 
to  be  reasonable  for  a  wide  range  of  prior  distributions  for  the 
particular  specified  criterion  levels  and  loss  ratios.    For  example, 
with  a  itq  value  of  .70,  and  a  prior  distribution  having  expectation  of 
.70.  the  general  recommendat:ion  for  a  loss  ratio  of  1.5  is  a  test  of 
eigtit  items  with  the  requirement  of  six  out  of  eight  correct  for 
advancement.    For  a  loss  ratio  of  2.0,  a  test  of  13  items  is  recommended 
with  a  score  of  ten  being  required  for  advancement.    The  ratio  of  2.5 
requires  a  test  of  14  items  with  eleven  correct  for  advancement,  and 
a  loss  ratio  of  3.0  requires  a  test  of  15  items  with  12  correct  for 
advancement. 


We  suspect  that  these  particular  recommendations  will  be  pleasant 
news  for  IPI  people.    If  the  loss  ratio  is  as  small  as  1.5,  as  it  may 
well  be  in  some  situations,  the  indication  here  is  that  an  8-item 
test  will  be  satisfactory.    Even  with  a  loss  ratio  of  2.0,  a  13-item 
test  will  do.    Loss  ratios  of  2.5  and  3.0  do  not  call  for  greatly 
increased  test  lengths,  however  loss  ratios  as  high  as  this  may  indicate 
that  the  structure  of  the  unit  and  its  relationship  to  other  units 
could  profitably  be  reevaluated.    We  shall  discuss  this  and  similar 
questions  later  in  this  report.    With  a  iTq  value  of  .75,  and  a  prior 
distribution  with  expectation  .75,  and  a  loss  ratio  of  1.5,  the  recom- 
mended test  length  is  10  items  with  a  minimum  advancement  score  of  8. 
While  this  will  likely  be  thought  of  as  a  very  acceptable  test  length, 
the  situation  does  not  remain  as  favorable  when  the  loss  ratio  rises 
to  2.0.    Here  the  test  length  recommendation  is  for  a  test  of  25  items 
with  a  minimum  advancement  score  of  16.    As  is  indicated  later  in  the 
paper,  this  situation  can  be  improved  by  training  the  group  to  a 
higher  average  level  of  performance.    For  a  ttq  value  of  .80  and  a 
prior  expectation  of  .80,  reasonably  satisfactory  test  length  specifi- 
cations are  obtainable  provided  the  loss  ratio  does  not  exceed  2.0. 
Specifically,  for  a  loss  ratio  of  1.5,  a  seven-item  test  is  deemed 
adequate  with  a  minimum  advancement  score  of  six.    The  loss  ratio  of 
2.0  on  an  eight-item  test  will  be  adequate  with  a  minimum  advancement 
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score  of  seven*    With  loss  ratios  2,5  and  3,0,  tests  of  length  20 
and  22,  with  minimum  advancement  scores  o£  17  and  19,  respectively, 
are  recommended. 

When  the  situation  does  suggest  the  appropriateness  of  high  loss 
ratios,  a  considerable  reduction  in  the  required  test  length  can  be 
obtained  by  seeing,  to  it  that  the  means  of  the  prior  distribution  is 
rather  higher  than  the  specified  criterion  level.    For  example,  with  a 
tTq  value  of  ,80  and  a  prior  expectation  for  tt  of  ,85,  12  and  13-item 
tests  will  be  adequate  for  loss  ratios  of  2,5  and  3,0,    This  compares 
with  test  lengths  20  and  22  where  the  prior  expectation  of  tt  was  ,80, 
The  Hovick-Lewls  paper  contains  detailed  discussions  of  the  relation- 
ship of  the  various  input  variables  beyond  that  which  we  shall  not 
cover  here,  though  we  urge  readers  to  study  that  paper  carefully. 
We  shall  consider  here  only  some  of  the  broad  implications  of  this 
study  for  the  very  structure  of  Individually  Prescribed  Instruction, 
First  we  would  note  that  there  has  been  a  definite  tendency  in  ipi 
to  require  relatively  high  criterion  levels;  typically,  the  value 
,85  1*5  used.    One  might  well  speculate  whether  this  really  reflects 
a  perceived  need  for  a  high  criterion  level,  or  whether  it  is,  in 
fact,  a  function  of  a  high  loss  ratio  combined  with  a  desire  for  a 
short  test  length.    Only  when  we  get  to  the  point  tliat  required  loss 
ratios  and  criterion  levels  can  be  independently  evaluated  will  it 
be  possible  to  use  tables  such  as  the  ones  presented  in  this  paper. 
We  might  speculate^  and  indeed  hope,  that  for  most  situations  a 
specified  criterion  level  of  ,75  will  be  adequate,  but  that  a  loss 
ratio  greater  than  one,  possibly  1,5  or  2,0,  will  be  appropriate. 

The  tables  presented  in  this  paper  also  have  a  great  implication 
for  the  amount  of  time  that  might  best  be  put  into  an  individual 
training  unit.    When  loss  ratios  are  high,  it  may  well  be  highly 
advantageous  to  strengthen  the  training  program  to  the  extent  that 
the  mean  output  is  well  above  the  specified  criterion  level.  This 
will  make  it  possible  to  use  short  tests,  or  alternatively,  will 
generally  reduce  die  risk  of  incorrect  classification.    This  will  of 
course  be  more  expensive  and  this  investment  must  be  balanced  out 
against  the  reduction  of  cost  of  testing  and  the  reduction  in  the 
expected  loss  due  to  incorrect  decision. 

Another  implication  of  these  tables  is  that  the  training  module 
should  also  be  structured  so  that  very  high  loss  ratios  are  not 
appropriate.    This  will  be  accomplished  by  seeing  to  it  that  individual 
modules  are  not  overly  dependent  on  preceding  ones.    Here  again,  one 
is  balancing  off  changes  in  the  module  itself  against  changes  in  the 
criterion-referenced  testing. 

We  would  empiiasize  again  that  the  primary  purpose  of  this  paper 
is  to  provide  a  structure  for  an  intelligent  discussion  of  decision 
making  within  IPI,  including  the  question  of  prescribing  test  length, 
The  results  contained  here,* we  think  will  be  useful,  but  tliey  should 
in  no  way  be  considered  to  be  definitive.    We  do  not  know  what  loss 
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distribution  of  the  percentage  of  items  answered  correctly  by  students, 
and  it  is  thus  possible  to  make  inferences  about  the  true  level  of 
functioning  of  each  student  and  the  mean  and  standard  deviation  of 
these  true  values  in  the  population  of  students*    It  is  further  pointed 
out  that  if  tlie  instructional  program  were  completely  efficient  and 
the  students  were  without  hutoan  frailty,  there  would  be  no  variation 
in  true  level  of  functioning  of  students  x>n  posttests.    A  student 
would  remain  in  a  unit  only  until  that  instant  at  which  his  level  of 
functioning  attains  a  prespecified  criterion*    However  nothing 
approaching  this  is  possible  with  present  instruction  technology.  In 
the  real  world  of  IPI  there  will  be  some  variation  of  true  levels  of 
functioning  among  students  on  posttests»    And  it  is  pointed  out  that 
this  background  information  can  be  combined  with  direct  observational 
information  to  improve  the  decision-making  process.    It  is  also  noted 
that  other  background  information  can  be  used,  namely  that  involving 
the  performance  of  the  student  on  tests  of  other  skills*    It  is  argued 
that  surely  a  person  scoring  highly  on  t^l  subtests,  and  a  little 
less  highly  on  t^th  would,  we  suspect,  have  a  true  score  on  the  t^th 
test  higher  than  his  observed  score,  and  that  this  somewhat  lower 
observed  score  might  be  due  in  part  to  bad  luck  or  carelessness*  It 
is  pointed  out  that  the  method  for  performing  these  analyses  is 
available  from  the  work  of  Wang  and  Lewis  (1973;  Appendix  3*5  and  3,6)* 

The  report  ends  by  illustrating  the  kind  of  format  that  can  be 
used  to  transmit  information  to  the  instructional  manager*  Currently 
following  an  IFI  posttest,  the  manager  receives  a  skill  profile  on 
each  student*    On  this  profile  the  percentage  correct  that  the  student 
got  on  each  of  the  skills  is  reported.    And  from  this  the  instructional 
manager  decides  which  skills  the  student  must  redo.    Under  the 
proposed  change  the  posttest  profile  would  not  consist  of  these 
percentage  correct  scores,  but  rather,  for  each  skill,  the  probability 
that  the  student's  true  level  of  functioning  is  greater  than  the 
specified  criterion  level.    Thus  while  the  information  to  be  fed 
to  the  instructional  manager  is  somewhat  different  than  it  has  been 
in  the  past,  it  is  certainly  no  more  complex.    It  will  be  necessary 
to  teach  instructional  managers  what  these  new  numbers  are,  what  they 
mean,  and  ho\7  they  are  to  be  used*    But  this  process  should  not 
be  terribly  difficult*    Indeed  there  may  well  be  enough  instructional 
material  in  this  particular  report  to  accomplish  this  task* 

The  fourth  paper  appearing  in  Appendix  Number  1  is  "A  Review  of 
Testing  and  Decision-Making  Procedures  for  Selected  Individualized 
Instructional  Programs"  by  Hambleton  (1973)*    While  the  current 
research  effort  lias  been  directed  primarily  at  one  particular 
individualized  program,  namely  IPI,  our  view  is  that  these  same 
methods  can  be  used  in  other  programs,  namely  Project  PLAN  and  the 
Mastery  Learning  Program,  as  well  as  in  various  approaches  to  computer- 
assisted  instruction.    The  survey  by  Hambleton  gives  us  enough  of  a 
picture  of  each  of  these  programs  to  confirm  this  belief,  and 
furthermore  suggests  that  none  of  these  programs  currently  has  any 
sort  of  well"developed  decision  process*    Undoubtedly  a  major  under- 
taking would  be  required  to  implement  the  Bayesian  decision-theoretic 
system  in  each  of  these*    There  is  no  question  in  our  mind  but 
that  this  would  be  useful, 
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6.    Statistical  Decision  Theory  for  Individually  Prescribed 
Instruction 

The  papers  in  Section  A  have  proposed  a  statistical  monitoring 
system  for  IPI  in  the  framework  of  statistical  decision  theory*  In 
particular*  a  threshold  loss  structure  for  the  problem  was  proposed 
and  the  implications  of  that  loss  structure  were  investigated*  At 
the  same  time  it  was  pointed  out  that  this  is  only  one  of  many 
possible  loss  structures  for  this  problem  and  that  Indeed  it  should 
only  be  considered  as  a  first  approximation  to  an  appropriate  loss 
structure* 

In  Appendix  2  we  present  a  paper  entitled  "A  Primer  on  Decision 
Analysis  for  IPI*',    The  purpose  of  this  paper  is  to  provide  a  brief 
seml^technical  presentation  of  decision  theory  set  within  the  context 
of  I?I»  so  that  persons  concerned  with  IPI  can  learn  enough  of 
decision  theory  to  understand  how  such  methods  can  be  applied*  The 
paper  begins  with  a  formal  statement  of  decision  theory  in  a  two 
decision  situation  and  demonstrates  the  application  of  normal  and 
extensive  form  analysis.    Computations  for  both  types  of  analysis 
are  presented  in  complete  detail  so  that  the  reader  can  see  how  each 
operates*    It  is  then  pointed  out  that  under  general  conditions*  both 
normal  and  extensive  form  of  analysis  will  always  lead  to  the  same 
decision*  and  since  extensive  form  analysis  is  the  easier  to  do>  it 
therefore  becomes  possible  and  desirable  to  adopt  it  as  the  standard 
procedure* 

In  extensive  form  analysis  the  first  task  is  to  compute  the 
posterior  probability  distribution  of  the  unknown  parameter  given  the 
prior  distribution  and  the  data.    Once  this  is  done  this  probability 
distribution  can  be  combined  with  the  statement  of  the  loss  structure 
to  arrive  at  an  extensive  form  decision*    The  discussion  then  turns 
to  the  use  of  extensive  form  analysis  with  continuous  posterior 
distributions  and  specific  applications  and  examples  relevant  to 
IPI  are  given*    This  continuous  form  for  the  prior  distribution  is  the 
one  that  has  been  discussed  previously  in  the  expository  papers* 
Particular  emphasis  is  placed  in  the  discussion  here  on  procedures 
for  determining  cutting  scores*  that  is  the  point  in  the  observed 
score  continut*'in  above  which  a  student  should  be  deemed  a  master  and 
below  which  he  should  be  deemed  a  nonmaster.    This  complements  the 
work  in  Appendix  1*1, 

There  then  follows  a  reasonably  complete  discussion  of  utility 
theory  which  indeed  makes  a  generalization  of  threshold  loss  possible* 
In  this  way  it  is  possible  to  have  different  utility  for  true  positive 
and  true  negative  decisions*    The  paper  then  discusses  a  linear 
utility  function  which  may  be  useful  in  some  situations  followed  by 
a  discussion  of  quadratic  utility  and  exponential  utility. 

The  most  reasonable  appearing  utility  function  and  one  which  may 
Indeed  be  most  appropriate  for  IPI  application  is  the  squared 
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exponential  utility  function*    A  brief  description  of  this  utility 
function  is  given*    There  then  follows  a  discussion  of  a  three  action 
problem  in  which  the  possible  decisions  are 


(1)  The  student  should  be  sant  tacl^  one  module^ 

(2)  He  should  be  retained  in  the  present  module^ 

(3)  He  should  be  advanced  to  the  next  module* 


A  complete  analysis  for  this  situation  is  discussed  within  the 
framework  of  threshold  utility*    It  is  then  discussed  Jn  the 
framework  of  linear  utility* 

The  final  section  of  the  report  deals  briefly  with  the  question 
of  deciding  whether  or  not  it  is  useful,  at  a  particular  point  in  a 
sequential  testing  environment,  to  take  another  observation^  that  is 
have  the  student  answer  another  item* 

It  is  a  general  property  of  Bayesian  inference  that  the  analysis 
is  identical  whether  observations  are  taken  all  at  unce  or  taken 
sequentially  with  the  possibility  of  stopping  whenever  a  decision 
can  legitimately  be  made*    For  example,  if  the  appropriate  rule  is 
that  a  student  can  be  advanced  if  he  gets  ceven  out  of  eight  and  the 
items  are  being  administered  sequentially,  he  could  terminate  test 
taking  if  he  were  to  answer  the  first  seven  items  correctly  or  if, 
at  any  time,  his  total  of  incorrect  responses  exceeded  one*    In  a 
more  general  framework  when  exact  Bayesian  solutions  are  used  rather 
than  the  approximate  ones  discussed  here  and  in  Appendix  1*1,  the 
formal  procedure  for  deciding  whether  or  not  an  additional  item 
should  be  administered  involves  comparing  the  expected  value  of 
information  to  be  obtained  from  that  item  with  the  cost  of 
administering  this  item*    This,  of  course,  is  that  the  value  of 
information  and  the  cost  of  administering  an  item  have  been  put  on 
a  common  scale*    In  practice  this  is  a  very  difficult  thing  to  do* 
We  have  not  In  this  report,  therefore »  considered  costs  Involved  in 
testing  but  In  theory  this  could  be  done* 
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It  my^  In  fact^  be  necessary  to  treat  repeaters  as  a  separate  groups 
These  considerations  can  of  course  only  be  determined  by  studying 
data  from  a  particular  IPI  application*    Nevertheless,  It  is  important 
that  this  assumption  be  well  understood  by  those  attempting  application 
of  this  technique* 

Appendix  Number  3*5  and  3*6  by  Wang  and  Lewis  (1973  a,  b), 
represents  a  significant  advance  In  statistical  technology  for  IPI 
monitoring*    While  the  theory  contained  In  these  papers  Is  complete, 
it  has  not  been  subject  to  even  the  modest  empirical  study  that  the 
simpler  theory  has  been,  and  as  a  result  we  do  not  have  as  clear  an 
Indication  at  this  point  as  to  what  further  practical  Improvement*^ 
on  the  simpler  theory  are  made  possible  here*    Tlie  Idea  behind  the 
work  In  these  references  Is  that  It  Is  possible  to  gain  Information 
about  a  particular  student's  ability  on  a  particular  behavioral 
objective  not  only  from  the  fact  that  he  was  In  an  Instructional 
program  with  a  group  of  other  students,  all  of  whom  received  the  same 
training  and  that  therefore  they  can  be  expected  to  be  at  a  roughly 
similar  level,  but  also  it  is  possible  to  note  that  this  student  has 
been  trained  and  is  now  being  tested  on  other  behavioral  objectives 
at  the  same  time,  and  that  his  ability  on  the  t-th  behavioral  objective 
will  surely  be  reflected  to  some  extent  by  his  perfcrmance  on  the 
other  t-1  objectives*    This  will  be  true  if  there  is  any  relationship 
at  all  between  the  t  objectives,  as  there  typically  is*    Thus  the 
whole  Kelley  (1923)  approach  can  be  used  to  adopt  collateral  information 
from  these  other  objectives  for  estimation  of  the  t-th  objective  on 
each  student* 

The  mathematical  approach  here,  following  toot  arcsl^ie  transformation, 
is  to  utilize  a  full  two-way  analysis  of  variance  model,  which  in  the 
first  Instance  was  studied  with  possible  interaction*    The  first 
Wang  and  lewis  paper  (1973  a)  shows  how  to  analyze  this  model,  and 
shows  how  to  get  out  point  estimates*    The  work  here  is  in  a  way 
similar  to  some  previous  work  done  by  Llndley  though  the  special- 
ization here  yields  much  more  simple  results  dua  to  the  fact  that  the 
variances  known* 

In  these  papers,  a  two-way  no-lnteractlcn  model  is  also  used  and 
it  is  found  that  on  the  data  sets  investigated  almost  no  difference 
in  point  estimates  were  obtained  from  the  interaction  and  no  inter- 
action models,    A  possibly  wild  speculation  here  1*  that  the  arcslne 
transformation  not  only  gives  homogeneous  variance,  and  to  oomL  excent 
normality  of  distribution,  but  also  has  the  desirable  side  effect  of 
tending  to  yield  addltlvlty* 

In  the  second  Wang  and  Lewis  paper  (1973  b),  the  no-interact ion 
model  is  employed  in  order  to  get  out  marginal  distributions  for  the 

in  a  manner  very  similar  to  that  used  in  Appendix  3,2,  These 
posterior  marginal  distributions  of  the  ^±  can  then  be  used  In  a 
decision  analysis  in  precisely  the  same  way  as  posterior  distributions 
from  simpler  analyses.    It  proved  impossible  to  obtain  marginal 
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distributions  numerically  with  the  Interaction  ntodel  and  so  the 
no-lnteractlon  model  was  used. 

The  work  In  Appendix  Number  3.5  and  3*6  should  be  useful  both 
In  posttest  and  placement  applications  where  Inference  needs  to  be 
made  simultaneously  on  several  behavioral  objectives  for  each  student. 
From  the  work  In  Appendix  3.1  we  have  a  reasonably  good  Idea  of  the 
benefit  to  be  gained  from  the  collateral  Information  on  other  students 
taking  the  test.    However^  we  have  not  as  yet  been  able  to  give 
sufficient  thought  to  have  any  Idea  of  how  much  Information  Is  gained 
and  how  much  resultant  decrease  In  testing  can  be  accomplished  by 
using  information  on  the  t-1  other  behavioral  objectives  for  each 
student.    In  any  event  there  Is  some  limitation  here^  In  that  In  order 
to  use  the  arcslne  transformation  theory  with  a  normality  assumption^ 
a  sample  of  eight  items  seems  to  be  almost  necessary  if  we  are  to  work 
at  ^11  away  from  the  center  of  the  distribution.    Thus  even  with  a 

value  of  .75>,  an  n  of  8  seems  desirable^  if  not  absolutely  necessary^ 
while  with  a  tTq  value  of  .85,  an  item  sample  of  size  8  seems  at  best 
to  be  barely  adequate  to  justify  the  assumptions  of  the  model. 

At  this  point  it  seems  clear  that  the  theory  will  be  extrem^sly 
useful,  but  that  it  will  be  necessary  to  study  and  determine  the  kinds 
of  distributions  that  are  to  be  found  in  practice  and  to  determine 
as  a  result  of  this  what  test  lengths  and  decision  rules  will  be 
appropriate.    The  discussion  in  Appendix  1.1  barely  begins  to  tap 
the  question  of  test  length  specification.    However,  it  will  be 
exceedingly  difficult  to  do  any  further  work  on  this  without  a  close 
look  at  real  data. 
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the  sophisticated  features  of  that  dialect,  and  to  then  write  programs 
which  could  be  very  easily  translated  into  any  other  dialect.  At 
present  we  have  been  working  with  the  Hewlett  Packard  HP2000C  dialect 
of  BASIC,  but  w<*  could  equally  well  work  with  the  Digital  Equipment 
Corporation  PDPll/40  dialect.    In  either  case,  if  care  is  taken  in 
the  writing  of  the  program,  translation  to  the  other  dialect  or 
indeed  to  any  other  dialects  would  be  very  easy* 

The  second  report  written  by  Isaacs  (1973,  Appendix  4.3),  carefully 
surveys  all  of  the  BASIC  dialects  available  in  March  of  1973,  and 
indicates  the  strength  and  weakness  of  each  of  these.    Tite  conclusion 
is  that  most  dialects  have  sufficient  capability  for  CADA  application, 
and  that  therefore  IPI  monitoring  could  be  accomplished  with  them. 
It  should  be  noted  that  BASIC  dialects  for  mini-computers  are  under 
constant  states  of  revision  and  that  by  the  time  this  report  is  filed, 
many  of  the  surveyed  dialects  will  be  much  stronger  than  they  were 
in  March  of  1973,    In  particular,  we  would  note  that:  the  Wang  2000 
super  desk  calculator  would  seem  to  come  very  close  to  having  the 
capability  for  CADA  application.    If  this  is  true,  then  the  cost  of 
the  computerized  IPI  management  becomes  almost  trivial. 

The  final  paper  in  this  report  is  a  technical  description  of  the 
CADA  monitor  indicating  how  subroutines  can  be  chained  to  the  monitor 
In  BASIC  and  how  it  is  possible  to  continually  update  and  improve 
the  monitor  without  disturbing  the  system  in  operation. 

The  statistical  methodology  developed  here,  the  availability  of 
relatively  inexpensive  computational  machinery,  and  the  clear  under- 
standing and  explication  of  a  theoretical  structure  for  IPI,  we  think 
now  makes  it  clearly  possible  and  highly  desirable  to  introduce  a 
structured  management  for  IPI.    The  theory  presented  here  is  still 
just  that,  theory.    To  make  it  work  it  is  now  necessary  to  implement 
these  decision-making  procedures  within  an  ongoing  IPI  operation. 
No  doubt  such  application  will  result  in  the  refinement  of  the  theory 
and  hopefully  In  its  improvement. 
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Introduction 


In  a  program  of  Individually  Prescribed  Instruction  (IPI),  where  a 
student's  progress  through  each  level  of  a  program  of  study  is  governed  by 
his  performance  on  a  test  dealing  with  individual  behavioral  objectives, 
there    is  considerable  value  in  keeping  the  number  of  items  on  each  test 
at  a  miniiRum.    The  specified  test  length  for  each  objective  must»  however^ 
be  adequate  to  provide  sufficient  information  regarding  the  student ^s  degree 
of  mastery  of  the  behavioral  objective  being  tested.    Just  what  the  minimum 
acceptable  length  will  be  depends  on  the  manner  in  which  test  information 
is  used  to  make  decisions  about  individual  students »  the  level  of 
functioning  required  for  defining  mastery  of  an  objective^  the  relative  losses 
Incurred  in  making  false  positive  and  false  negative  decisions^  the  background 
information  available  on  the  student  and  on  the  instructional  process^  and 
the  premiiim  on  testing  time  within  the  instructional  process.    Our  purpose  in 
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OEG"0'72"0711  with  the  Office  of  Education,  U.  S.  Department  of  Health, 
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Contractors  undertaking  such  projects  under  Government  sponsorship  are 
encouraged  to  express  freely  their  professional  judgment  in  the  conduct 
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in  Evaluation^  Number  3,  a  publication  supported  in  part  by  the  National 
Institute  of  Education  and  by  the  American  Educational  Research  Association. 
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this  paper  Is  to  discuss  these  Issues  and  provide  some  broad  guidelines  for 
test- length  specification  for  IPX  posttestsi    These  specifications  will  be 
tentative  because  of  unresolved  substantive  and  methodological  lssu3s»  but 
we  believe  that  they  should  provide  some  Improvement  on  current  practice* 
A  separate^  and  rather  more  complex  treatment  will  be  required  for  placement 
and  pretest  length  specification* 

Background 

In  a  criterion-referenced  measurement  approach  to  Individually 
Prescribed  Instruction^  we  Imagine  a  population  of  test  Items,  having  mixed 
Item  difficulty,  dealing  with  a  particular  objective  and  an  Ideal  decision 
which  advances  a  student  past  this  objective  If  he  Is  able  to  answer  at  least 
a  given  percentage  of  the  items  in  the  population*    This  minimum  passing 
percentage,  the  so-called  criterion  level»  simply  reflects  the  degree  of 
mastery  deemed  sufficient  for  this  objective  (although  it  implicitly  involx^es 
the  difficulty  of  the  items  as  well)*    The  actual  percentage  of  items  that 
a  person  would  answer  correctly  in  the  population  of  items  is  called  his 
level  of  functioning*    In  practice^  the  advancement-retention  decision  must 
be  maae  from  a  small  sample  of  observations  (test  items),  and^  hence,  errors 
in  the  decision  process  must  be  expected* 

One  common  treatment  of  the  test  length  problem  in  a  criterion- 
referenced  measurement  context  has  been  given  by  Mlllman  (1972)*  He 
studied  a  standard  decision  rule  which  advances  the  student  if  the 
percent  of  items  correctly  answered  on  a  test  equals  or  exceeds  the 
required  criterion  level*    Here  it  is  assumed  that  the  Items  on  the  test 
may  be  treated  as  a  random  sample  from  the  population  of  Interest,  so 
that  the  obtained  percentage  correct  is  a  useful  estimate  of  the  true 
population  percentage  for  the  student*    Using  binomial  probability 
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Table  1 

Percent  of  Students  Expected  To  Be  Incorrectly 
Advanced  or  Retained 


Specif lea  Criterion  Level  .70 


Advancement       No*  of 


Student *s  True  Level  of  Functioning* 


Score 

Test  Items 

50 

55 

60 

65  i 

70 

75 

80 

85 

90 

95 

6 

7 

6 

10 

16 

23  ' 

67 

55 

42 

28 

15 

4 

6 

8 

15 

22 

32 

A3 

45 

32 

20 

11 

4 

1 

7 

9 

9 

15 

23 

34 

54 

AO 

26 

14 

5 

1 

7 

10 

17 

27 

38 

51  ' 

35 

22 

12 

5 

1 

a 

11 

11 

19 

30 

A3 

A3 

29 

16 

7 

2 

9 

12 

7 

13 

23 

35 

51 

35 

20 

9 

3 

10 

13 

5 

9 

17 

28 

58 

42 

25 

12 

3 

n 

14 

3 

6 

12 

22  ' 

64 

48 

30 

15 

4 

12 

15 

2 

4 

9 

17  ! 

70 

54 

35 

18 

6 

Specified  Criterion  Level 

Student *s  True  Level  of  Functioning* 


Advancement 
Score 

No*  of 
Test  Items 

50 

55 

60 

65 

™l 

75 

80 

85 

90 

95 

6 

8 

15 

22 

32 

43 

55  1 

32 

20 

11 

4 

1 

7 

9 

9 

15 

23 

34 

46 

40 

26 

14 

5 

1 

8 

10 

6 

10 

17 

26 

38 

47 

32 

18 

7 

1 

9 

11 

3 

7 

12 

20 

31 

55 

38 

22 

9 

2 

9 

12 

7 

13 

23 

35 

49 

35 

20 

9 

3 

16 

20 

1 

2 

5 

12 

24 

58 

37 

17 

4 

17 

21 

1 

4 

9 

20 

63 

41 

20 

5 

18 

22 

1 

3 

7 

17  ' 

68 

46 

23 

6 
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Table  1  (continued) 


Specified  Criterion  Level  >80 


Student's  True  Level  of  Functioning* 


Ad  va  n  c  cinen  t 
Score 

No.  of 
Test  Items 

5r 

55 

60 

65 

70 

75  ' 

80 

85 

90 

&5 

6 

7 

6 

10 

16 

23 

33 

A5  1 

A2 

28 

15 

A 

7 

8 

A 

7 

11 

17 

26 

37  1 

50 

3A 

19 

6 

8 

9 

2 

A 

7 

12 

20 

30  1 

56 

AO 

23 

7 

8 

10 

6 

10 

17 

26 

38 

53  ' 

32 

18 

7 

1 

9 

11 

3 

7 

12 

20 

31 

A6  1 

38 

22 

9 

2 

10 

12 

2 

A 

8 

15 

25 

39  ' 

AA 

26 

11 

2 

11 

13 

1 

3 

6 

11 

20 

33  1 

50 

31 

13 

2 

12 

15 

2 

A 

9 

17 

30 

A6  1 

35 

18 

6 

17 

20 

1 

2 

A 

11 

23  ' 

59 

35 

13 

2 

19 

22 

1 

3 

7 

16  ' 

67 

A2 

17 

2 

Specified  Criterion  Level  >85 


Student^s  True  Level  of  Functioning* 


Advancement 
Score 

No.  of 
Test  Items 

50 

55 

60 

65 

70 

75 

80  1 

85 

90 

95 

7 

8 

A 

7 

11 

17 

26 

37 

50  1 

3A 

19 

6 

8 

9 

2 

A 

7 

12 

20 

30 

AA  1 

AO 

23 

7 

9 

10 

1 

2 

5 

9 

15 

2A 

38  1 

A6 

26 

9 

10 

11 

1 

1 

3 

6 

11 

20 

32  1 

51 

30 

10 

11 

12 

M 

1 

2 

A 

9 

16 

28  1 

56 

3A 

12 

17 

19 

M 

1 

2 

5 

11 

24  1 

56 

29 

7 

19 

21 

M 

1 

3 

8 

18  1 

63 

35 

8 

*The  true  level  of  functioning  Is  the  percent  of  Items  a  student 
would  be  able  to  answer  correctly  If  he  were  given  the  entire  universe 
of  It ems » 

Students  having  true  level  of  fur*ctlonlng  values  less  than  the  specified 
criterion  level  should  fall  a  test  composed  of  all  Items  from  this  universe* 
However,  on  any  given  test  of  finite  length,  some  of  these  students  will  get 
more  than  the  minimum  advancement  percent  of  the  Items  correct  and  be 
considered  as  *'passers"*    The  expected  percent  of  such  Incorrect  advancements 
are  given  In  the  body  of  the  table  to  the  le^t  of  the  dotted  line* 

Students  having  true  level  of  functioning  values  equal  to  or  greater 
than  the  minimum  advancement  percent  should  pass  such  a  test.    The  percent 
of  these  students  who  will  be  Incorrectly  retained  are  shown  In  the  table 
to  the  right  of  the  dotted  line. 
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tables,  Mlllman  obtained  the  probability  that  a  student  with  a  given 
true  level  of  functioning  wo»jld  be  incorrectly  advanced  or  retained  by 
this  procedure. 

Table  1  expands  on  some  of  Millman's  conjputations  and  gives  the 
conditional  probability  of  incorrect  advancement  or  retention  tor  a  variety 
of  true  levels,  test  lengths,  and  minimum  passing  percentages.    The  first 
impression  this  table  provides  is  that  a  substantial  proportion  (sometimes 
more  th.^n  half)  of  tJie  students  with  true  levels  close  to,  or  at  the 
criterion  level,  will  be  incorrectly  advanced  or  retained,  at  least  for 
the  test  lengths  considered.    There  appears  to  be  a  slight  improvement 
in  accuracy  of  decision  as  the  test  length  increases  from  8  to  22  items, 
although  this  effect  is  largely  hidden  by  fluctuation  in  the  probabilities, 
due  to  changes  in  the  percentage  correct  required  for  advancement.    For  examp].e, 
with  a  criterion  level  of  ,7,  the  percentage  correct  required  for  advancement 
is  ,75,  '78,  ,70,  ,73,  or  ,75  for  test  lengths  of  8,  9,  10,  11,  or  12  items, 
respectively.    This  brings  up  a  question  as  to  the  optimality  of  the  decision 
procedure  assumed  in  Table  1,    To  provide  a  framemrk  for  answering  this 
question,  let  us  consider  some  of  the  issues  involved. 

Suppose  seven  out  of  eight  were  taken  as  the  minimum  advancement  score 
when  the  criterion  level  os  .75;  the  probability  of  incorrect  advancement 
would  decrease  substantially  for  all  students  with  true  levels  below 
the  criterion  level.    This  is  shown  in  Table  2,    On  the  other  hand, 
those  above  • 75  suffer  a  substantial  increase  in  their  chances  of  being 
incorrectly  retained.    Apparently,  a  more  general  framework  is  required 
before  even  the  decision  procedure  can  be  chosen,  much  less  any  judgment 
made  concerning  minimum  test  length.    This  framework  would  need  to  tuke 
into  account  on  which  side  of  ,75  small  expected  errors  were  considered 
to  be  more  important. 
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Table  2 

Percent  of  Students  Expected  To  Be  Incorrectly 
Advanced  or  Retained 
Criterion  Level  «  ,75   Test  Length  «  8 


Advancement 
Score 


True  Level 


50     55     60     65  7C 


75     80     85     90  95 


6 
7 


15  22 
4  7 


32  43  55 
11     11  26 


32  20  11  4  1 
63     50     34     19  6 


A  Framework  For  Specifying  Test  Length 
Table  1  is  very  helpful  in  identifying  the  seriousness  of  the  problem 
of  short  tests.    From  a  practical  poirtt  of  view*  however,  a  solution  to  the 
problem  must  involve  looking  at  a  different  conditional  probability »  and 
abandoning  the  simple  decision  procedure  that  Millman  has  so  convincingly 
demonstrated  to  be  inadequate.    Instead  of  the  probability  that  a  student 
will  attain  a  particular  test  score»  given  his  true  levels  it  is  the 
probability  that  a  student^s  true  level  of  functioning  exceeds  the  specified 
criterion  levels  given  his  test  score»  which  is  required  in  making  a  decision. 
In  other  words,  it  is  the  test  score™not  the  true  level — which  is  given 
(i,e,  observed),  and  which  is  the  basis  for  any  decision  to  advance  or 
retain  the  student.    Thus,  a  student  should  be  advanced  only  if  the  probability 
that  he  has  attained  or  surpassed  the  criterion  level,  given  his  test  score* 
is  sufficiently  high.    To  obtain  the  necessary  probability,  an  application 
of  Bayes  theorem  is  required.    In  such  an  analysis,  prior  knowledge 
(expressed  in  prob^bJ  liiiti:  c^?rms)  of  the  student^s  true  level  of  functioning 
is  combined  with  the  (binomial)  model  information  relating  the  observ.^'.d 
test  score  to  true  level;  and»  tihe  re£ -rt  is  a  posterior  probabiJ^ity 
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distribution  for  true  level  of  functioning,  given  test  score.  The 
probability  this  distribution  assigns  to  levels  above  the  criterion 
is  the  quantity  of  interest.    In  this  f orm*alation,  the  problem  can  be 
described  as  selecting  a  minimum  ^^ample  size  and  an  advancement  score,  so 
that  students  attaining  that  score  will  then  have  a  sufficiently  high 
pr-rbability  of  having  at  least  the  rainimuin  required  level  of  functioning. 

As  a  first  approximation,  let  us  suppose  our  Imowledge  of  a  jj^tudent^s 
true  level  of  functioning  is  vague,  prior  to  having  his  test  results. 
If  this  state  of  knowledge  is  characterized  by  selecting  a  uniform 
distribution  on  the  interval  from  zero  to  unity  for  true  level,  ir^  Bayes 
theorem  provides  the  Posterior  probabilities  listed  in  Table  3  for  various 
scores  and  test  lengths.    The  posterior  distributions  on  which  these 
probabilities  are  based  all  belong  to  the  Beta  family,  and  the  parameters 
in  each  case  are  those  given  in  the  table,  primarily  for  future  reference. 

To  generate  a  decision  procedure  on  the  basis  of  Table  3,  we 
must  select  a  criterion  level  (ir^)  and  a  minimum  acceptable  probability 
that  a  student's  true  level  (ir)  exceeds  this  criterion.    Thus,  for  example, 
we  might  take  ir^  =  .80  and  the  minimum  acceptable  Prob(ir  >^  '^^W*  ^)  "  '50, 
where  x  is  test  score  and  n  is  test  length.    We  would  then  be  saying  that 
we  wanted  to  advance  the  student  only  if  we  were  at  least  50% 
sure  that  his  level  of  functioning  was  above  ,80.  Thon,  using  Table  3, 
we  see  that  with  n  =^  8,  all  students  having  x  >^  7  would  advance  to  the 
next  objective,  but  not  those  with  x  =  6,    For  a  test  of  12  items,  the 
minimum  advancement  score  would  be  10  correct. 

Note,  however,  that  if  we  required  80%  assurance  that  the  true  level 
of  functioning  was  above  '80,  [Prob(TT  >^  .80)  >^  *80],  then  even  those  with 
eleven  correct  responses  to  twelve  items  would  not  be  advanced.    We  think 
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Table  3 

Probability  Student's  True  Level  Of  Functioning  Is 
Greater  Than  it   Given  A  Uniform  Prior  Distribution 


Mlnintum 
Advancement 
Score 

No.  of  Posterior 
Test  Items  Distribution 

50 

55 

Criterion 
60    65  70 

Level — IT 
75  80 

o 

yu 

95 

6 

8 

6(7,  3) 

91 

85 

77 

66 

54 

40 

26 

14 

5 

1 

7 

8 

6(8,  2) 

98 

96 

93 

88 

80 

70 

56 

40 

23 

7 

8 

8 

6(9,  1) 

100 

100 

99 

98 

96 

92 

87 

77 

61 

37 

7 

9 

6(8,  3) 

95 

90 

83 

74 

62 

47 

32 

18 

7 

1 

8 

9 

6(9,  2) 

99 

98 

95 

91 

85 

76 

62 

46 

26 

9 

9 

9 

6(10,  1) 

100 

100 

99 

99 

97 

94 

89 

80 

65 

40 

7 

10 

B(8,  4) 

89 

81 

70 

57 

ii3 

29 

16 

7 

2 

8 

10 

6(9,  3) 

97 

93 

88 

80 

69 

54 

38 

22 

9 

2 

9 

10 

6(10,  2) 

99 

S9 

97 

94 

89 

80 

68 

51 

30 

10 

8 

11 

6(9,  4) 

93 

87 

77 

65 

51 

35 

21 

9 

3 

9 

11 

6(10,  3) 

98 

96 

S2 

85 

75 

61 

26 

11 

2 

10 

11 

0(11,  2) 

100 

99 

9S 

96 

92 

84 

73 

56 

34 

12 

9 

12 

6(10,  4) 

95 

91 

83 

72 

58 

A2 

25 

12 

3 

10 

12 

6(11,  3) 

99 

97. 

94 

89 

80 

67 

50 

31 

13 

2 

11 

12 

6(12,  2) 

100 

100 

99 

97 

94 

87 

77 

60 

38 

14 
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that  it  is  unreasonable  to  require  perfect  performance  as  a  standard  for 
advancement,  and  therefore,  we  need  to  improve  upon  this  analysis.  One 
way  is  to  use  a  longer  test,  but  we  can,  at  least,  hope  to  find  a  procedure 
in  which  a  twelve**item  test  will  be  adequate. 

The  results  in  Table  3,  although  they  provide  relevant  information 
for  mastery  decisions  about  students  based  on  test  scores,  do  not 
take  full  advantage  o.""  the  power  which  is  available  through  the  use 
of  prior  knowledge,    xn  p^^rticitlar,  it  will  seldom  be  the  case  that  our 
knowledge  of  a  student *r;  crue  level  is  adequately  described  by  a  uniform 
distribution.    For  example,  our  prior  probability  that  a  student  is 
functioning  above  a  criterion  level  of  ,8  might  be  approximately*  ,75. 
This  would  be  the  case  if  historical  data  suggested  ttiat  about  757*  of 
the  students  who  completed  a  unit  of  Individually  Prescribed  Instruction 
proved  to  be  at  or  above  mastery  level.    Moreover,  we  might  judge  the 
str^^rigth  of  our  knowledge  to  be  roughly  equivalent  to  that  based  on  a 
score  from  a  12-item  test.     (A  method  for  making  this  assesment  will  be 
referenced  shortly,) 

When  working  with  a  binomial  model,  it  is  convenient  and  generally 
very  satisfactory  to  select  a  member  of  the  Beta  class  of  distributions  to 
characterize  prior  beliefs  (Novick  and  Jackson,  1974).    If  this  is  done,  the 
posterior  distribution  is  easily  obtained,  and  in  every  instance  will  again 
be  a  member  of  the  Beta  family.    In  fact,  if  the  prior  distribution  is 
3(a,  b)  and  x  success  in  n  trials  are  observed,  then  the  posterior  distri- 
bution is  6(x  +  a,  n  -  X  +  b).    This  can  be  seen  in  Table  3,  where  it  is 
noted  that  the  uniform  distribution  is  6(1,  1),    If  we  i  strict  ourselves 
to  prior  distributions  In  Che  Beta  family,  the  beliefs  specified  in  the 
previous  paragraph  are  characterized  by  6(10,254,  1,746),    Given  this  prior 
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distribution  and  the  Indicated  test  results,  the  posterior  distributions 
and  posterior  probabilities  of  exceeding  various  criteria  are  provided  In 
Table  4,    The  precise  stipulation  of  prior  distributions  must  always  be 
done  carefully,  but  extensive  aids  (Novlck  and  Jackson,  1974,  Novlck, 
Lewis,  and  Jackson,  1973)  are  available,  and  Indeed  an  elaborate  system 
of  Computer  Assisted  Data  Analysis  (CADA)  Is  available  (Novlck,  1973)  to 
help  an  instructional  decision  maker  specify  his  prior  distribution,    A  yet 
more  sophisticated  way  of  getting  prior  and  posterior  distributions  for 
each  person  Is  derived  by  Lewis,  Wang,  and  Novlck  (1973J  and  the  required 
tables  are  given  by  Wang  (1973),    Por  the  present,  we  shall  suppose  that 
this  work  has  been  done  carefully  and  that  the  prior  distribution  used  in 
the  construction  of  Table  4  is  appropriate. 

Tables  3  and  4  demonstrate  clearly  the  Impact  of  prior  knowledge 
on  our  Interpretation  of  test  results.    In  Table  3,  for  example,  the 
posterior  probability  that  a  student  with  a  score  of  six  out  of  eight 
items  correct  has  a  true  level  greater  than  ,80  is  only  .26,  whereas 
in  Table  4  this  probability  has  Increased  to  .60.    This  result  should  not 
be  surprising,  in  view  of  the  fact  that  we  have  now  set  this  probability 
to  be  .75,  aprlorl  as  compared  to  ,20  in  Table  3.    If  we  felt  the  chances 
to  be  very  good  that  the  student  had  mastered  an  objective  (to  a  level  above 
.8)  before  we  saw  the  test  results,  then  a  score  of  six  out  of  eight  will 
not  substantially  change  our  beliefs;  it  will  lower  the  probability,  but 
aposterlorl  may  still  leave  the  odds  in  favor  of  mastery.    In  many 
applications,  a  prior  probability  of  mastery  may  be  no  more  than  •60,  but 
the  results  will  still  differ  sharply  from  those  obtained,  assuming  vague 
prior  information.    Note  that  if  we  were  to  adopt  the  rule  that  Je  will 
advance  a  student  if  the  aposterlorl  probability  of  mastery  is  at  least 
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Table  4 

Probability  Student's  True  Level  of  Functioning  Is 

Greater  Than  it  Given  A  6(10,254,  1,746)  Prior  Distribution 
  0  


Minimum 

Advancement       No,  of  Posterior 
Score        Test  Items  Distribution 


Criterion  Level — 
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95 

86 
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1 

87 

72 

48 

22 
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96 

88 

72 

45 

13 

76 

55 

30 

10 

1 

89 

75 

52 

24 

4 

96 

90 

75 

48 

14 

6 
7 
8 
7 
8 
9 
7 
8 
9 
b 
9 
10 
9 
10 
11 


8  ^(16.254,  3.746nOO  100    98  96  90 

8  6(17.254,  2.746)100  100  100  99  97 

8  6(18.254,  1.746)100  100  100  100  99 

9  6(17.254,  3.746)100  100  99  97  92 
9  6(18.254,  2.746)100  100  100  99  98 
9  6(19.254,  1.746)100  100  100  100  100 

10  6(17.254,  4.746)100    99    97  93  84 

10  6(18.254,  3.746)100  100    99  98  93 

10  6(19.254,  2.746)100  100  100  99  98 

11  6(18.254,  4.746)100  99  98  94  87 
11  6(19.254,  3.746)100  100  100  98  95 

11  6(20.254,  2.746)100  100  100  100  99 

12  6(19.254,  4.746)100  100  J9  96  89 
12  6(20.254,  3.746)100  100  100  99  96 
12  6(21.254,  2.746)100  100  100  100  99 


,746)|100 

Note:    The  mean  and  mode,  respectively  of  6(10.254,  1.746)  are 
.855  and  .925   and  for  this  distribution  Prober  >  u^)  for       =  .70,  .75, 
.80,  .85  are  .92,  .86,  .75,  and  .59,  respectively.    A  close  look  at  these 
distributional  characteristics  will  help  a  decision  maker  determine  If 
this  prior  distribution  is  a  realistic  characterization  of  his  beliefs. 
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*5Qf  then  In  this  example^  we  will  advance  him  If  the  prior  distribution 
were  that  of  Table  4,  but  not  If  It  were  that  of  Table  3* 

When  the  decision  maker  specifies  an  Informative  prior  distribution, 
he  Is  saying.  In  effect,  that  he  wants  a  decision  which  will  have  a  high 
probability  of  being  correct  In  that  portion  of  the  decision  space  In  which 
he  thinks  the  student's  ability  truly  lles»    For  example,  referring  to 
Table  2,  a  decision  mker  with  a  high  prior  probability  that  the  student  had 
a  true  level  of  functioning  below  ,75  would,  by  virtue  of  his  analysis, 
require  a  minimum  passing  score  of  seven  correct  out  of  eight  ltems»  This 
would  assure  him  a  low  probability  of  mlsclasslf icatlon  for  all  values 
below  ^75*    Another  decision  maker  with  high  prior  probability  that  the 
student  was  above  criterion  level  would  likely  require  only  six  out  of 
eight  correct,  and  thus  have  low  probability  of  an  Incorrect  decision  for 
values  of  »75  or  above* 

Once  we  have  decided  to  work  with  the  posterior  probability  that  a 
student's  level  of  functioning  exceeds  some  criterion,  given  his  test 
score,  and  have  made  use  of  our  prior  knowledge  In  obtaining  this 
probability,  another  Issue  remains  to  be  settled  before  we  can  turn 
to  the  question  of  test  length*    Simply  sta:ed,  we  need  to  know  how  sure 
we  should  be  that  a  student  has  mastered  an  objective  at  the  chosen  level 
beiuib  we  make  the  decision  to  allow  him  to  advance  to  the  next  objective* 
For  Instance,  Is  a  posterior  probability  of  at  least  »5,  as  was  used  In 
the  last  example,  a  reasonable  choice  In  all  cases?    AlmDst  certainly 
this  last  question  should  be  answered  In  the  negative*    The  point  at 
Issue  here  comes  down  to  an  understanding  of  the  relative  disutilities  or 
losses  associated  with  the  false  positive  and  false  negative  errors* 
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If  It  were  no  more  serious  to  advance  a  student  whose  level  was  below 
the  criterion  than  to  retain  a  student  who  was  above^  we  would  be  behaving 
optlntally  If  we  were  to  advance  students  with  posterior  probabilities  above 
.5  and  retain  the  others.    In  many  situations  the  prior  probability  will  be 
this  hlgh^  and  hence  an  advancement  decision  could  then  be  made  on  an  aprlori 
basis.    On  the  other  hand>  we  might  consider  the  loss  to  be  twice  as  great 
for  a  false  advancement  than  for  a  false  retention.    In  this  case>  we  should 
only  advance  those  students  whose  posterior  probability  for  being  above  the 
criterion  exceeds  2/3,    The  general  result  is  that  we  shall  achieve  the 
smallest  expected  loss  if  we  match  the  posterior  odds  to  the  loss  ratio, 
Thusj  If  the  loss  ratio  is  2  to  1  (false  advance  to  false  retain) >  a 
probability  of  2/(2  +  1)  gives  matching  odds  of  2/3  to  l/S  above  criterion  to 
below  criterion). 

Table  5 

Losses  Associates  With  Incorrect  Decisions 


Tl  >  Tl 

*-  o 


True  Level 

Tl   <  Tl 


Decision 


Advance 
Kataln 


0 

a 

b 

0 

To  express  the  result  symbolically^  consider  the  notation  of  Table  5, 
Here  a  is  the  loss  associated  with  advancing  a  student  whose  true  level  is 
below  Ti^>  and     Is  the  loss  for  retaining  a  student  whose  true  level  exceeds 
Ti^,    The  decision  rule  which  minimizes  expected  loss  In  this  situation  is 
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to  advance  a  student  if  his  test  score  is  such  that 

b  Prob(Tr  >_  ir^lx,  n)  >_  a  Prob(Tr  <  ir^jx,  n), 

and  to  retain  him  otherwise.    This  comparison  is  equivalent  to  comparing 

the  loss  ratio  a/b  to  the  probability  ratio  Prob(Tr  >^Tr^|x,  n)/Prob(ir  <  ir^jx,  n 
If  a  ~  b  in  our  analysis^  the  decision  procedure  reduces  to  comparing 

the  median  of  the  posterior  distribution  with  the  specified  criterion 

level.    If  the  median  is  at  least  at  this  levels  the  student  is  advanced^ 

otherwise  he  is  retained.    In  this  situation^  the  decision  procedure  is 

very  similar  to  that  used  by  Millman  (1972).    Though  the  procedure  used  by 

Millman  is  not  Bayesian^  it  is  equivalent  to  comparing  with  the  mode  (rather 

than  the  median)  of  the  posterior  distribution  based  on  a  uniform  prior. 

Thus,  in  effect,  the  sampling  theory  approach  gives  equal  weight  to  all 

equal  intervals  throughout  the  range  of  ir;  that  is,  effectively,  to  take  ir  to 

be  uniformly  distributed  apriori.    This  is  seldom  a  reasonable  prior 
specification.    We  might  also  remark  that  the  formulation  in  Table  5  can  be 

generalized  to  provide  for  differential  utilities  for  correctly  identifying 

true  positives  and  true  negatives  as  well  ^s  differential  disutilities 

(or  losses)  for  false  positives  and  false  negatives  as  is  done  in  Table  5, 

To  do  this  negative  quantities  (negative  disutilities  =  utilities)  would 

need  to  replace  the  zeros  in  Table  5»  and  a  slightly  more  complicated 

analysis  would  not  be  used. 

It  may  be  worthwhile  to  summarize  the  situation  at  this  point.  An 

instructor  wishing  to  use  test  results  in  the  context  of  Individually 

Prescribed  Instruction  should  be  ready  to  supply  three  kinds  of  information. 

First,  a  criterion  level — the  minimum  degree  of  mastery  required — must  be 

set.    In  Individually  Prescribed  Instruction  this  seems  to  run  from  about 
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,70  to  about  ,85,    Second^  prior  knowledge  of  the  student's  true  level  of 
functioning  must  be  translated  Into  probability  terms ^  namely  a  prior 
probability  distribution  for  ^  ,    Typically,  a  carefully  monitored  program 
will  be  such  as  to  suggest  a  prior  probability  distribution  that  assigns  a 
prouablllty  of  just  more  than  ,50  to  the  region  above  the  criterion  level. 
If  this  Is  not  the  case,  the  general  efficacy  of  the  program  should  be 
re-evaluated,    A  program  that  results  In  a  much  higher  probability  may  be 
wastefully  long  and  one  that  results  In  a  lower  probability  may  require 
strengthening.    Finally,  the  relative  losses  associated  with  the  two  types 
of  Incorrect  decisions  must  be  assessed,    A  ratio  of  more  than  1/1  Is  the 
rule  (we  are  told)  with  ratios  of  1,5/1  and  2/1  being  common,  and  ratios 
as  high  as  3/1  not  being  rare. 

It  should  be  clear  that  all  three  of  the  above  determinations  will 
have  an  Influence  on  the  minimum  necessary  test  length.    As  the  criterion 
level  approaches  unity,  the  test  must  be  longer  In  order  to  provide  adequate 
Information  about  a  student's  level  of  functioning  in  the  neighborhood 
of  the  criterion.    If  prior  probabilities  of  mastery  are  sufficiently  high, 
very  short  tests  become  possible,  but  this  Is  not  and  should  not  be  the 
typical  case.    Finally,  higher  loss  ratios  require  longer  tests  to  allow 
the  possibility  of  high  posterior  probability  of  mastery.    We  shall  also 
see  that  greater  test  lengths  are  sometimes  required  because  of  the  obvious 
restriction  to  Integer  valued  sample  sizes, 

A  Design  For  Test-Length  Specification 
The  characteristics  of  the  group  of  students  being  tested  must  now 
be  considered  as  they  relate  to  test-length  specification.    Each  member 
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Table  6 

Selected  Prior  Distributions  For  IPI  Advancement  Decisions 


Prior 
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15 
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.18 

.23 
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20 

$(13.5,  1.5) 

15 
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*Note:    All  entries  have  been  rounded  to  two  decimal  places  and  smoothed  so  that  the 
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of  the  group  of  students  tested  has  been  exposed  to  the  same  Instruction 
program  under  Identical  local  conditions*    If  a  particular  student  is 
not  considered  atypical  for  this  groups  then  our  prior  beliefs  about  his 
true  level  of  functioning  should  closely  reflect  the  true  distribution 
of  levels  of  functioning  found  In  that  group*    Indeed,  elaborate  formal 
procedures  for,  effectively,  bootstrapping  a  prior  distribution  using, 
for  each  examinee,  the  scores  on  the  remaining  m  -  1  examinees  are 
described  by  Novlck,  Lewis,  and  Jackson  (1973)*    Thus,  a  group  characteristics 
through  their  effect  on  our  prior  distributions,  do  affect  test-length 
specification*    If  the  average  test  score  of  the  group  Is  high  (l*e*, 
above  the  criterion  level)  and  there  Is  little  variation  among  Individuals, 
shorter  tests  become  feasible* 

Since,  In  practice,  prior  distributions  will  be  based  upon  on-site 
experience,  there  will,  of  course,  be  different  prior  distributions 
for  different  sites*    Uhat  we  shall  attempt  to  do  here  Is  to  show  what 
sample  sizes  will  be  required  for  a  broad  range  of  prior  distributions 
and  loss  ratios*    What  we  need  to  do  now,  therefore.  Is  to  consider  certain 
combinations  of  prior  distributions,  criterion  levels  and  loss  ratios, 
and  see  what  sample  size  lilll  be  adequate  In  each  case* 

Frr  our  analyses,  we  shall  consider  20  different  prior  distributions 
for  the  level  of  functioning  tt^  four  specified  criterion  levels,  and  four 
loss  ratios*    For  each  criterion  level,  we  shall  consider  all  four  loss 
ratios  and  four  of  the  prior  distributions*    The  four  loss  ratios  we 
shall  use  are  1*5,  2*0,  2*5,  and  3*0*    The  respective  probabilities 
P  «  Prob(Tr  >^  TT^)  required  for  advancement  (given  by  setting  P/(l  -  P) 
equal  to  the  loss  ratios,  a/b]  are  *60,  *67,  *71,  and  *75*    Thus,  with  a 


52 


18 


loss  ratio  of  3*0,  the  posterior  probability  that  the  student's  level  of 

functioning  is  greater  than  the  specified  criterion  level  must  be  at  least 

*75,  if  he  is  to  be  advanced* 

The  twenty  prior  probability  distributions  we  shall  be  considering 

are  given  in  Table  6  where  they  have  been  grouped  in  blocks  of  five,  with 

each  block  having  a  distribution  with  the  respective  mean  values  •70,  •TS, 

,80,  *85>  and  *90*    The  blocks  differ  with  respact  to  the  concentration  of 

the  prior  distributions*    Within  block,  the  distributions  differ  with 

respect  to  their  mean  values*    Note  that  in  the  first  block  the  arguments 

of  each  Beta  distribution  sum  to  8,  e*g*,  5*6  +  2*4  ■  8*    This  indicates 
that  the  amount  of  prior  information  contained  in  each  of  these  distributions 

is  equivalent  to  what  would  be  gained  from  a  test  containing  eight  items*  If 

given  one  of  these  prior  distributions  and  some  criterion  level  and  loss  ratio, 
we  specify  an  eight— item  test»  our  posterior  distribution  will  contain 

information  equivalent  to  that  contained  in  16  observations*    This  contrasts 

with  the  classical  procedure  which  uses  no  prior  information*    It  is  this 

increment  in  information  that  is  equivalent  to  prior  observations  which 

permits  a  reduction  in  test  length  when  a  Bayesian  procedure  is  used* 

The  first  problem  in  doing  an  analysis  is  that  of  selecting  a  reasonable 

prior  distribution.    For  the  present  application,  we  would  first  need  to 

ask  ourselves  what  we  would  expect  to  find  as  the  mean  level  of  functioning 

in  our  p,^sttest  group*    With  a  apecified  criterion  level  of  *70,  we  might 

hope  for  a  mean  level  of  functioning  of  *70*    Thus,  we  would  have  people  in 

training  until  such  time  as  we  would  "expect**  them  to  be  qualified*  Since 

loss  ratios  are  typically  greater  than  one,  some  overtraining  may  be  thought 

to  be  useful,  but  as  we  shall  see,  excessive  overtraining  may  be  wasteful* 
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Suppose^  for  concreteness^  that  we  believe  the  mean  population  level 
of  functioning  to  be  ,70*    Distributions  1>  6,  11,  and  16  satisfy  this 
condition,  and,  hence,  we  may  choose  from  among  these»    We  note  that 
these  distributions  are  in  an  increasing  order  of  tightness,  as  may  most 
conveniently  be  seen  in  the  probability  assignment  given  in  the  last  column, 
to  the  interval  (-90,  1*00)*    These  probabilities  are  respectively  *08, 
*05,  •OS,  and  •02,    We  need  to  ask  ourselves  which  of  thecs  v^ilues  seems 
most  reasonable,  and  this  then  will  give  us  some  preference  among  these 
prior  distributions*    We  might  consider  the  relative  weight  of  prior 
information  ar  Jmed  by  each  prior  distribution  (8,  10,  12,  and  15  equivalent 
prior  observations,  respectively),  and  this  should  help  to  narrow  our 
focus  to  one  or  two  adjacent  prior  distributions  for  this,  or  any  other 
application.    Since  the  authors  of  this  paper  cannot  know  what  an  appro- 
priate prior  distribution  will  be  in  applications  they  have  not  seen, 
it  will  be  most  helpful,  we  think,  to  work  out  sample  size  allocations 
for  several  prior  distributions  and  leave  the  final  selection  to  be  made 
"in  the  field".    We  believe  that  the  prior  distributions,  loss  ratios, 
and  specified  criterion  levels  u&^A  here  are  typical  of  those  found  in 
practice,  and,  therefore,  that  the  specific  results  we  shall  obtain  will 
be  useful.    However,  if  other  combinations  present  themselves,  we  believe 
that  the  general  methodology  that  we  are  demonstrating  should  be  adequate 
to  the  problem*    Actually  we  shall  f}.nd  that  most  of  our  specifications 
arc  very  robust  with  respect  to  the  choice  of  prior  distribution  within  the 
range  we  have  considered* 

Some  Specific  Test  Length  Recommendations 
In  Table  7,  we  give  recommended  sample  sizes  and  minimum  advancement 
scores  for       "  -70,  (a/b)  =  1,5,  2*0,  2*5,  3*0  and  prior  distributions 
1,  6,  11,  and  16,    The  values  that  we  have  settled  on  for  the  body  of 
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Table  7 

Recommended  Sample  Sizes  and  Advancement  Scores 


Tf    »  .70 

0 


Prior 


Loss  Ratio 


Distribution        (v)        1,5  (.60)         2.0  (.67)        2.5  (.71)        3.0  (.75) 


6(5.6,  2.4)'-  (.70)  6/8(.62) 
6(7,  3)  (.70)  6/8(.61) 

6(8.4,  3.6)       (.70)  6/8(.61) 


10/13 (.70)  11/14 (.74) 
10/13(.69)  11/14(.73) 
10/13(.68)  11/14(.72) 


6(10.5,  4.5)      (.70)        9/12(.62)^       10/13(.67)  11/14(.71) 


12/15 (.78) 
12/15 (.77) 
12/15 (.76) 
12/15(.75) 


6/8(75%) 


General  Recommendations 
10/13(77%)       11/14(79%)  12/15(80%) 


"^Apriori,  Prob(Tt  >^  .70)  for  each  of  the  four  prior  distributions  is 
.54,  .54,  .53,  and  .53. 

^For  6/8,  Prob(Tt  >  .70)  -  .598. 
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this  table  are  not»  in  every  instance^  optimum  in  any  statistical  sense» 
though  we  are  confident  that  the  risks  associated  with  these  decision  rules 
are  in  every  case  insignificantly  different  from  the  risks  of  the  optiwD 
procedures*    In  selecting  values  for  this  table  we  have  sought  sample 
sizes  and  minimum  advancement  scores  that  would  be  very  efficient  over 
a  wide  range  of  prior  distributions*    That  we  have  been  successful  in  this 
endeavor  is  confirmed  by  our  ability  to  give  general  recommendations 
'  that  hold  throughout  the  range  of  prior  distributions  studied*    Actually  in 
only  one  instance  have  we  cheated  (see  footnote  2,  Table  7),  but  again 
the  increase  in  expected  loss  will  be  trivial*    We  would  also  note  that 
the  required  percentage  correct  and  the  number  of  required  observations 
increases  as  the  loss  ratio  increases,  v;hich  "makes  sense'*  on  intuitive 
grounds 4 

A  rough  indication  of  the  near  optimality  of  any  of  the  individual 
specifications  can  be  gained  from  the  closeness  of  the  aposteriori 
probability  (indicated  in  parentheses  following  the  specification)  with 
the  value  required  by  the  particular  loss  ratio  (given  in  parentheses 
at  the  top  of  the  column).    Thus,  with  the  prior  distribution  P(7,  3),  the 
decision  rule  "six  out  of  eight",  abbreviated  6/8,  leads  to  the  aposteriori 
distribution  6(13,  5)  and  to  Prob(7r  >  •TO)  =  ,61  which  is  just  *01  greater 
than  the  required  level  *60  for  the  loss  ratio  1.5  (1*5  to  1)*    In  this 
instance,  the  specJ*:ied  decision  rule  may  be  very  good.    On  the  other 
hand,  consider  the  prior  distribution  6(5*6,  2,4),    Here  the  rule  11/14 
leads  to  a  value  •74  when  only  *71  is  required  for  a  2*5  to  1  loss  ratio* 
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Actually,  the  specification  8/10  is  somewhat  better  giving  a  posterior 
probability  of  *729*    Also  for  the  prior  distribution  3(7,  3),  the  posterior 
probability  with  8/10  is  *718*    With  the  loss  ratio  2.0/1  and  with  the 
prior  B(5*6,  2.4),  the  rule  7/9  leads  to  the  posterior  probability  *68  as 
compared  to  desired  value  of  *67*    In  every  case  where  we  have  specified 
an  **almost  best**  decision  rule,  the  result  has  been  an  increase  in  the 
specified  sample  size  and  the  purpose  has  been  to  obtain  uniformity  of 
specification  over  a  reasonably  wide  range  of  amounts  of  prior  information* 
Considering  our  general  ignorance  concerning  what  might  be  an  appropriate 
prior  distribution  in  specific  applications,  the  specifications  we  have 
given  should  be  the  more  generally  useful* 

Another  indication  of  how  good  a  particular  specification  is  can  be 
inferred  from  the  closeness  of  the  percentage  correct  required  by  the 
advancement  rule  to  the  specified  criterion  level*    Clearly,  if  the 
percentage  required  by  the  advancement  rule  is  very  much  larger  than  the 
specified  criterion  level,  a  large  percentage  of  qualified  students  will 
be  retained  and  this  is  undesirable,  particularly  for  small  loss  ratios. 
For  large  loss  ratios,  this  is  less  important  and  hence  higher  advancement 
ratios  can,  and  will  need  to  be  tolerated*    This  feature  is  exhibited  in 
Table  7,  where  the  advaticement  ratios  increase  with  increasing  loss  ratios. 
One  can,  of  coarse,  keep  the  advancement  ratio  down  very  close  to  the 
specified  criterion  level  even  for  higher  loss  ratios,  but  only  by  having  much 
larger  sample  sizes*    For  example  with  the  prior  distribution  3(5,6,  2*4) 
the  specified  criterion  level  tf^  «  *70  and  the  loss  ratio  2*0>  the  advancement 
ratio  72/100  is  satisfactory  since  Vxoh(K  >  *70l72/100)  ^  ,675,  but 
the  indicated  sample  size  is  unacceptable. 


ERIC 


57 


23 


ERIC 


Table  8 

Recommended  Sample  Sizes  and  Advancement  Scores 

TT    =  .75 


Prior 

Loss 

Ratio 

Distribution 

Cm 

1.5  (.60) 

2.0  (.67) 

2.5  (.71) 

3.0  (.75) 

6(6,  2)^ 

(.75) 

8/l0(.65) 

l6/20(.70) 

17/21(.74) 

l8/22(.77) 

6(7.5,  2.5) 

(.75) 

8/10 (.64) 

16/20 (.69) 

17/21(.73) 

18/22 (.76) 

6(9,  3) 

(.75) 

8/ 10 (.63) 

16/20 (.69) 

17/21(.72) 

l8/22(.75) 

6(11.25,  3.75) 

(.75) 

8/l0(.62) 

l6/20(.68) 

7/21(.7l) 

l9/23(.77)^ 

General  Reconmendations 

8/10(80%) 

16/20(80%) 

17/21(81%) 

18/22(82%) 

'^Apriori,  Proh(iT  >  .75)  ^  .56,  .55,  .55,  and  .54,  respectively,  for  the 
four  prior  distributions  used  in  Table  3. 

^For  18/22,  Prob(ii  >_  .75)  =  .744. 

Table  9 

Recoinmended  Sample  Sizes  and  Advancement  Scores 

IT    »  .80 

o 

Prior  .  ^'^^^ 

Distribution      (^M         1.5  (.60)        2.0  (.67)        2.5  (.7l)       3.0  (.75) 


6(6.4,  1.6)- 
6(8,  2) 
6(9.6,  2.4) 
6(12,  3) 


(.80) 
(.80) 
(.80) 
(.80) 


6/7(.66) 
6/7 (.65) 
6/7 (.64) 
6/7(.63) 

6/7(86%) 


7/8 (.70) 
7/8(.69) 
7/8 (.68) 
7/8(.67) 


17/20(.72) 
17/20(.72) 
17/20(.7l) 
1S/21(.73)' 


General  Recommendations 
7/8(88%)  17/20(85%) 


l9/22(.78) 
l9/22(.77) 
l9/22(.76) 
19/22 (.75) 

19/22(86%) 


priori,  Prob(ii  >_  .80)  »  .57;  for  8/10,  Prob(n  >_  .80)  =  .55;  for  16/20, 
Prob(it  >_  .80)  =  .54;  for  8.5/10,  Prob(ii      .80)  »  .67;  for  8.3/10, 
Prob(it  >_  .80)  =  .62;  for  9/10,  Prob(ii  >_  .80)  «  .78. 

^For  17/20,  Prob(it  >.  .80)  ^  .70. 
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Note  that  for  each  of  the  prior  probabilities  used  in  Table  7, 
Prob(Ti  >_  .70)  >  ,50.    Thus,  on  an  apriori  basis,  advancement  would  be 
indicated  with  a  loss  ratio  1,0,    This  will  generally  be  true  for  the  prior 
distributions  we  shall  be  adopting  for  our  analyses.    The  point  is  that 
loss  ratios  of  1,0  are  not  (we  are  told)  typical  of  IPX  applications,  and 
if  test  lengths  are  to  be  kept  reasonable  it  will  be  necessary  to  use 
training  programs  that  give  mean  output  at  or  above  the  criterion  level. 

There  has  been  a  definite  tendency  in  IPX  to  require  relatively  high 

advancement  ratios;  typically,  the  value  ,85  is  used.    One  might  well 

speculate  whether  this  is  a  function  of  a  high  loss  ratio  combined  with 

a  desire  for  a  short  test  length,  or  whether  it  really  reflects  a  perceived 

need  for  a  high  criterion  level,     (For  example  an  advancement  ratio  of  6/7 

with  the  prior  distribution  3(5,6,  2,4)  would  yield  with  x  -=  6  a  posterior 

Prob(iT  >  -70)  =  ,77  which  would  bo  just  right  with  a  loss  ratio  of  3,0,) 

The  authors  of  this  paper  do  not  know  the  answer  to  this  question,  but  hope 

that  those  within  XPI  will  want  to  consider  it  carefully.    Only  through 

such  serious  consideration  can  the  test  length  problem  be  ^'solved". 

Some  recommended  test  lengths  for  '75  and  four  prior  distributions 

with  £7(1^)  =  4  75  are  given  in  Table  8,    Again  we  have  been  able  to  specify 

one  generally  satisfactory  advancement  ratio  for  each  of  the  four  loss 

ratios.    We  note  that  the  required  test  lengths  for       =  ,75  are  rather 

larger  than  for  it^  =  ,70,    Xn  Table  8,  we  find  very  short  required  test 

lengths  for  a  1,5  loss  ratio  and  rather  long  ones  for  loss  ratios  of  2,0, 

2,5,  and  3,0, 

Xn  Table  9,  we  provide  recommendations  for      *  ^80  when  Cc(it)  =  ,80, 
The  results  here  parallel  those  of  Table  8,  except  that  the  advancement 
ratios  are  very  high  as  compared  to  the  criterion  3.evels,    This  is 
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relatively  unsatisfactory.    In  Footnote  1  to  Table  9,  we  indicate  the  formal 
results  for  the  prior  distribution  B(6,4,  1,6)  and  the  sample  result 
"8,5"  correct  and  "1.5"  incorrect  and  also  for  "8,3"  correct  and  "1.7" 
Incorrect,    These  provide  very  nice  results  for  loss  ratios  of  2,0  and  1'5> 
respectively.    Unfortunately,  these  are  unobtainable  sample  results.  This 
demonstrates  that  in  part,  large  required  test  lengths  may  sometimes  be 
due  to  the  discreteness,  and  hence,  discontinuity  of  our  possible  experi" 
mental  outcomes.    This  also  suggests  that  the  precise  specification  of  the 
advancement  rules  may  be  highly  sensitive  to  the  mean  value  of  the  prior 
distribution  even  if  it  is  proving  to  be  relatively  insensitive  to  the 
total  amount  of  information  contained  in  the  prior  distribution,  which  is 
indicated  by  the  sura  of  the  two  parameters  of  the  Beta  distribution. 

For  example,  given  the  prior  distribution  6(6,4,  1,6)  and  the 
impossible  sample  result  x  ~  8,3,  n  =  10,  we  have  the  posterior  distri- 
bution 6(14,7,  3,3)  which,  as  we  indicated  previously,  gives 
Prob(Tr  >  ,80)  =  ,62  which  suggests  that  the  advancement  ratio  8,3/l0 
might  be  very  favorable  with  a  loss  ratio  of  1,5,    But  suppose  we  had 
just  a  slightly  different  prior  distribution,  namely,  ^(6,7,  1,3)  with 


(tt)  =  ,84,  then  the  sample  result  x  =  8,  n  *  10  would  yitld  the  posterior 
distribution  S(l4,7,  3,3)  and  thus,  for  the  reasons  gii^en  above,  indicate 
that  the  advancement  ratio  8/10  might  be  attractive.    This  advancement 
ratio  is  clearly  more  attractive  than  the  ratio  6/7,  despite  the  fact  that 
it  requires  three  additional  items,  because  this  ratio  8/l0     80%  is  closer 
to  the  criterion  level  than  is  the  advancement  ratio  6/7  "  86%, 

Because  of  this  relatively  high  dependence  of  the  results  on  the 
expected  value  of  the  prior  distribution,  it  seems  important  to  attempt 
some  study  of  the  variation  of  our  results  as  a  function  of  changes  in 
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Table  10 

Reconmended  Sample  Sizes  and  Advancement  Scores 

If   »  .80 


Prior 

I«oss 

Ratio 

Distribution 

1.5  <.60) 

2.0  <.67) 

2.5  <.71) 

3.0  <.75) 

5 

p\0«0>  L*£.) 

\'o5) 

Of  L\J\*VH) 

9/1K.69) 

10/12<,72)^ 

11  /l  O  ^ 

JLJL/JLJ^.  /Oj 

e<8.5,  1.5) 

<.85) 

8/l0<.66) 

9/1K.70) 

10/12 <. 73)^ 

11/13<.76) 

e<10.2,  1.8) 

<.85) 

8/l0<.67) 

9/1K.71) 

9/ll<.71)^ 

11/13<.77) 

e<12.75»  2.25) 

<.85) 

8/10<.69) 

9/1K.72) 

9/1K.72)* 

11/13 <. 78) 

General  Recoinnendatlnns 

8/l0<80%) 

9/ll<82%) 

10/12<83X) 

11/ 13 < 85%) 

■^For  5/6,  Prob<if  >_  .80)  =  .72. 

^For  5/6,  Prob<if  >  .80)  »  .73. 

^For  10/12.  Prob<if  >_  .80)  =  .74. 

Sor  10/12,  Prob<if  >,  .GO)  -  .75, 
5 

For  the  four  prior  distributions,  the  apriori  probabilities  of  tt  >_  .80 
are  .72,  .73,  .74,  and  .75.    With  these  prior  distributions  and  with  7/10, 
the  posterior  probabilities  of  ir     .80  are  .41,  .43,  .46,  and  .48. 
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our  prior  distribution*    For  this  reason,  we  have  in  Table  10  redone  our 
sample  size  recommendations  under  the  assump^-lt  n  that  the  mean  of  our 
prior  distribution  is  ,85  instead  of  ,80, 

Surely  the  practitioner  will  find  the  sample  size  recommendacions 
of  Table  10  to  be  attractive*    Apparently  with  these  prior  disbributions, 
test  lengths  need  be  no  greater  than  13  for  any  of  the  listed  loss-ratios* 
With  the  prior  distributions  having  ^(ir)  =  »80,  a  sample  size  of  22  is 
required  when  the  loss  ratio  is  3*0* 

What  is  happening  is  that  we  are  beginning  with  fairly  strong  beliefs 
that  If     TT^  so  that  not  much  data,  in  confirmation,  is  required  even  for 
high  loss  ratios*    In  fact,  even  on  an  apriori  basis,  an  advancement 
decision  would  be  made  for  all  loss  ratios  up  to  and  including  2*5* 
Indeed,  we  see  that  the  function  of  the  sample  data  here  is  to  provide 
the  possibility  of  obtaining  some  information  that  might  change  the 
decision  to  retention.    For  example,  an  observed  performance  ratio  of 
10/13  with  the  prior  distribution  6(6.8,  1.2)  would  give  aposteriori 
Prob(ir  >_  -80)  »  *72,  and  hence,  the  student  would  be  retained  if  the 
loss  ratio  were  3.0  (see  also  Footnote  5>  Table  10), 

We  believe  that  the  comparison  of  the  specifications  in  Tables  9 
and  10  have  important  implications  for  IPI  management*    When  loss  ratios 
are  high,  it  may  well  be  highly  advantageous  to  strengthen  the  training 
program  to  the  extent  that  the  mean  output  is  well  above  the  specified 
criterion  level*    This  will  make  it  possible  to  use  short  tests  or* 
alternatively  will  generally  reduce  the  risk  of  incorrect  classification* 
This  will,  of  course,  be  more  expensive,  and  this  investment  must  be  balanced 
out  against  the  reduction  in  the  cost  of  testing  and  the  reduction  in  the 
expected  loss  due  to  incorrect  decision*    The  final  Tabic,  Table  11>  looks 
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Table  11 

Recomnended  Sample  Sizes  and  Advancement  Scores 

IT   ■=  .85 
o 


Prior 

Loss 

Ratio 

Distributions 

{^(ir) 

1.5  (.60) 

2.0  (.67) 

2.5  (.70) 

3.0  (.75) 

3(6.8,  1.2)^ 

(.85) 

7/8(.62) 

9/ 10 (.70) 

17/19 (.73) 

18/20(.76)^ 

3(8.5,  1.5) 

(.85) 

7/8(.62) 

9/10 (.69) 

17/19 (.72) 

19/21(.77) 

3(10.2,  1.8) 

(.85) 

7/8(.61) 

9/10 (.68) 

17/19 (.72) 

19/21 (.76) 

3(12.75,  2.25) 

(.85) 

7/8 (.60) 

9/10(.67) 

17/19(.71)^ 

19/21 (.75) 

General  Recomnendations 

7/8(87.5%) 

9/10  (9o;s) 

17/19(89%) 

19/21(90%) 

^he  apriori  probabilities  for  it  >  .85  are  .59,  .58,  .58,  and  .57. 
^For  10/11,  Prob(ir  >  .85  =  .695). 
^For  19/21,  Prob(ir  >  .85  -  .78). 
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Very  much  like  Table  9  as  far  as  test  lengths  are  concerned*    Here  again 
some  robust  length  assignments  are  obtained^  though  agaln»  the  lengths  for 
the  high  loss  ratios  border  on  being  discomforting.    This  can  be  corrected 
by  training  to  an  average  level  of  functioning  of  *90*    With  the  prior 
distribution  B(7*2,  8),  we  find  that  Prob(Tr  >_  *85)  =  *76,  aprlorl*  Observing 
6/7  yields  Prob(Tr  >  ,85)  «  *70,  while  5/7  yields  a  value  of  *4I,  Observing 
8/9  yields  *77,  while  7/9  yields  *493*    Thus,  clearly,  very  short  test 
lengths  are  again  possible  If  the  students  are  trained  to  a  sufficiently 
high  average  standard. 

Some  Summary  Remarks 
The  test  length  recommendacJons  given  In  this  paper  are  meant  to  be 
taken  seriously  and  hopefully  they  will  soon  be  adopted  on  a  provisional  and 
experimental  basis,  so  that  more  experience  can  be  gained  vhlle  some  of 
the  theoretical  and  substantive  Issues  raised  In  this  paper  ar^  debated*  The 
questions  of  level  of  functioning  required  to  define  mastery  and  the 
relative  losses  Incurred  In  making  false  positives  and  false  negative  decisions 
require  serious  discussion  and  concensus*    We  also  need  to  get  some  clear 
picture  of  what  kinds  of  distributions  of  outcomes  are  to  be  expected  as  this 
determines  the  amount       prior  information  available  In  making  individual 
assessments*    This  third  Issue  Is,  as  we  have  Indicated,  intimately  related 
to  the  expected  level  of  functioning  that  Is  sought  in  the  group  being  trained* 
Hopeful  and  possible  outcomes  of  such  discussions  could  be  a  consensus  that: 
1,    In  most  situations  a  level  of  functioning  of  something  less  than 
,85  is  satisfactory.    A  value  as  low  as  .75  would  be  highly 
desirable.    This  could  be  accomplished  by  redefining  the  task 
domain  slightly  to  eliminate  Very  easy  Items* 
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2»    Training  should  be  carefully  monitored  so  that  expected  group 
performance  will  be  just  slightly  higher  than  the  specified 
criterion  levels    This  will  keep  training  time  and  testing  time 
relatively  low* 

3»    The  program  should  be  structured  so  that  very  high  loss  ratios  are 
not  appropriate.    That  Is  to  say,  Individual  modules  should  not 
be  overly  dependent  on  preceding  ones* 

One  problem  that  does  not  arise  with  Bayeslan  methods  Is  any  complication 
If  sequential  methods  are  used*    Items  can  simply  b$  atimlnlstered  until 
It  Is  clear  that  a  student  will  definitely,  or  cannot  possibly,  attain  the 
minimum  advancement  score*    Thus  with  a  minimum  advancement  score  o£  8/10, 
testing  can  cease  as  soon  as  light  successes  or  three  failures  are  observed. 

Two  Issues  have  been  treated  In  a  rather  gross  way  In  this  paper  and 
on  these  Important  Issues  further  research  needs  to  be  done.    First  It 
must  be  recognized  that  while  the  threshold  loss  function  we  have  adopted 
here  Is  a  better  approximation  to  reality  than,  for  example.  Livings t on 
criterion  centered  squared*^ error  loss  (see  Hambleton  and  Novlcic,  1973), 
It  Is  only  a  gross  approximation  to  be  used  while  better  and  more  complicated 
approximations  are  being  Investigated.    Three  thai  Immediately  come  to  mind 
are: 

1>  A  threshold  loss  function  with  an  Indifference  region  In  which 
there  Is  zero  loss  for  false  positive  or  false  negative  errors* 

2*  A  negative  squared-exponentlal  loss  used  with  the  root  arcslne 
transformation  parameter 

Y  *  sln'^vV 
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3*    A  cumulative  Beta  distribution  loss  function. 
We  expect  that  these  loss  functions  will  give  somewhat  different  and  surely 
better  length  specifications  than  those  obtained  here»  but  the  overall 
decrease  in  expected  loss  may  or  may  not  be  great*    We  should  also  remark 
that  these  recommendations  are  specifically  made  for  first  time  through 
decisions*    We  have  yet  to  consider  the  problem  of  decisions  for  students 
repeating  a  unit. 

Finally^  we  would  remark  that  one  of  the  important  issues  that  we 
identified  at  the  outset  of  this  paper  has  been  handled  in  a  most  casual 
and  informal  manner*    To  do  other  than  this  would  have  enormously  complicated 
the  analysis  and  delayed  substantially  the  appearance  of  our  recommendations > 
We  refer  explicitly  to  the  premium  on  testing  time  within  the  instructional 
process  and  implicitly  to  an  implied  trade-off  between  training  and  testing 
time.    A  completely  general  analysis  would  consider  an  available  time  T  and 
an  allocation  of  T  into  instruction  and  testing  times  i  +  t  «       so  as  to 
maxiiT-ize  a  payoff  function  which  would  have  a  (possibly  differential)  positive 
payoff  for  each  module  successfully  completed^  and  a  (differential)  negative 
payoff  for  an  incorrect  decision  of  either  type*    We  are  reluctant  to  undertake 
such  a  sophisticated  analysis  until  such  time  as  the  operating  conditions 
of  IPI  are  more  clearly  defined . 

For  the  present  paper  we  have  implicitly  adopted  some  guidelines  which 
effectively  say  that  it  Is  very  desirable  to  have  test  lengths  of  12  or 
less,  tolerable  but  undesirable  to  have  test  lengths  as  high  as  20  and 
discomforting  to  have  tests  that  are  longer  than  this*    We  have  also  taken 
the  position  that  a  decision  should  not  be  made  on  che  basis  of  prior  and 
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collateral  information  alone  but  that  mastery  must  be  confirmed  by  a  test 
that  permits  demonstration  o£  nonmastery.    As  In  all  of  the  judgmental 
decisions  made  In  this  paper  we  have  been  guided  by  counsel  from  experienced 
IPI  personnel,  particularly  Richard  Ferguson  and  Anthony  Nltko  to  whom 
we  are  much  Indebted.    The  value  of  this  paper  will  largely  be  determined 
by  the  quality  of  the  discussion  engendered  by  It  among  such  people. 
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ABSTRACT 


The  decision  process  required  for  Individually  Prescribed  Instruction  (IPI),  an  adaptive 
instructional  program  developed  at  the  University  of  Pittsburgh,  is  described  In  IPI,  shorl  tests 
are  used  to  determine  the  level  of  proficiency  of  each  student  in  precisely  defined  learning 
objectives.  The  output  of  these  tests  is  used  to  guide  instructional  planning  for  individual 
students. 

The  nature  and  effect  of  errors  in  proficiency  decisions  are  described  and  a  procedure  for 
reducing  the  probability  of  such  errois  is  proposed.  The  plan  calls  for  a  Bayesian  procedure 
which  would  incorporate  prior  information  on  the  instructional  program,  for  example  the 
distribution  of  the  percentage  of  items  answered  correctly  by  students  Such  a  procedure  would 
permit  inferences  about  the  true  level  of  functioning  of  each  student. 

The  final  section  of  the  paper  proposes  tv/o  methods  for  implementing  these  procedures  in  an 
ongoing  IPl  program,  one  approach  calls  for  the  integration  of  the  procedure  as  a  part  ol  a 
computer-based  instructional  management  system*  whereas  the  second  approach  describes 
how  the  procedure  can  be  made  tractable  in  a  typical  non-automated  IPl  classroom 


Prepared  by  the  Research  and  DevGlopniGnt  Division 

The  American  College  Testing  Proprain 

For  additional  copies  write: 
Publications  Division 
Tho  American  College  Testing  Program 
P,0,  Box  168,  Iowa  City»  Iowa  62240 


(Check  or  money  order  must  accompany  request.)  Price;  $1.00 

;r!c  72 


IMPLEMENTATION  OF  A  BAYESIAN  SYSTEM  FOR  DECISION  ANALYSIS 
IN  A  PROGRAM  OF  INDIVIDUALLY  PRESCRIBED  INSTRUCTION 


The  research  reported  herein  was  performed  pursuant  toGrant  No.  OEG-0-72-071 1  with  the  Office  of  Education. 
U.S.  Department  of  Health,  Education,  and  Welfare,  Melvin  R.  Novick,  Principal  investigator  Contractors 
underlaking  such  profects  under  Government  sponsorship  are  encouraged  to  express  freely  their  professional 
judgment  in  theconduct  of  the  project.  Points  of  view  or  opinions  stated  do  not,  therefore,  necessarily  represeni 
official  Office  of  Education  position  or  policy. 


73 


IMPLEMENTATION  OF  A  BAYESIAN  SYSTEM  FOR  DECISION  ANALYSIS 
IN  A  PROGRAM  OF  SNDIVIDUALLY  PRESCRIBED  INSTRUCTION 


Richard  L.  Ferguson 
Metvin  R.  Novick 


INTRODUCTION 


The  leasibility  of  instructional  programs  designed 
to  adapt  to  the  individual  needs  of  learners  has  been 
adequately  demonslrated  by  educational  systems 
like  Individually  Prescribed  Instruction  (Glaser. 
1S68)  and  A  Program  for  Learning  in  Accordance 
with  Needs  {Hanagan,  1967).  Although  these 
programs  accomplish  individualization  insomewhat 
different  ways*  each  includes  components  which 
can  be  described  by  the  following  sequence  of 
operations: 

1.  Specification  of  the  learning  objectives  in  terms 
of  observable  student  behavior 

2.  Assessment  of  the  student's  entering  compe- 
tencies. 

3.  Assignment  or  election  of  educational  material 
and/or  experiences  fitted  to  the  student^s 
individual  needs. 

4.  Continuous  assessment  and  monitoring  of  the* 
student^s  performance  and  progress 

Since programslikelPI  and  PLAN  callforadaptation 
of  the  learning  environment  to  meet  individual 
requirements,  they  necessarily  rely  heavily  on  the 
systematic  assessment  of  sludent  progress.  Glaser 
{1 968)  has  observed  thai,  in  iPL  test  data  serve  as  the 
primary  source  of  information  enabling  teachers  to 
make  differential  decisions  regarding  student 


instruction.  Thus,  steps  (2)  and  (4)  play  a  prominent 
role  in  the  successful  implementation  of  IPI  A 
review  of  current  decision-making  procedures  lor 
lour  selected  individualized  instructional  programs 
has  been  given  by  Hambleton  (1973) 

The  fundamental  purpose  for  testing  in  in- 
dividualized instructional  programs  hke  IPl  and 
PLAN  IS  to  ascertain  whether  or  not  the  student  has 
attained  some  prescribed  level  of  proficiency  In  a 
specified  learning  objective,  Hambleton  and  Novick 
(1973)  have  observed  that,  "Questions  of  precise 
achievement  levels  and  comparisons  among  indi- 
viduals on  these  levels  seem  to  be  largely  irrel- 
evant," Because  test  data  are  used  initially  to  place  a 
student  at  the  appropriate  pomt  within  an 
instructional  program  or  sequence,  and  thus  to 
identify  appropriate  learning  materials  or  ex- 
periences given  his  needs,  the  lest  models  which 
have  emerged  to  serve  this  function  are  very  dif- 
ferent from  those  used  for  standard  instructional 
models.  Because  these  tests  relate  a  student's  per- 
formance on  items  drawn  from  a  carefully  specified 
domain  toa  prespecified  criterion  or  standard,  these 
tests  have  come  to  be  called  domain  or  cntenon- 
referenced  tests. 

It  is  not  the  purpose  of  thrs  paper  to  contrast  the 
differences  between  norm- referenced  tests  and 
criterion- referenced  tests  Suffice  it  to  say  that 
criterion-referenced  tests  are  deliberately  con- 
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structed  so  as  to  yield  measurements  which  are  di- 
rectly rnterpretable  in  terms  of  specified  per- 
formance standards  (Glaser  and  Nitko,  1971).  The 
process  of  constructing  such  tests  involves  the 
specification  of  a  domain  of  tasks  that  the  student 
should  be  able  to  perlorm  and  the  selection  of 
samples  of  these  tasks  representative  of  that  do- 
main. The  students  competency  in  the  skill  is 
judged  in  terms  of  hrs  performance  in  respondingto 
the  sample  of  the  tasks  which  is  drawn.  Performance 
on  this  sample  is  used  to  infer  that  his  level  of 
functioning  in  the  domain  either  does  or  does  not 
meet  some  prescribed  standard. 

Because  student  performance  on  tests  used  in  IPI 
and  PLAN  is  used  as  the  basis  for  making  decisions 
affecting  placement  and  advancement. and  because 
it  is  crucial  that  these  decisions  be  accurate,  major 
importance  is  attached  to  the  precision  with  which 
each  person's  true  domain  score  (level  of 
functioning)  can  be  related  to  the  prescribed 
proficiency  level  However,  due  to  time  constraints, 
the  tests  are  often  comprised  of  a  very  smad  number 
of  items,  usually  10  or  less.  Thus,  the  precision  of 
judgment  from  such  tests  must  be  open  toquestion. 
Because  of  the  important  role  which  testing  plays  in 
the  instructional  decision  making  within  IPI,  im- 
provement in  the  quality  of  the  decision  process 
would  be  greeted  with  considerable  enthusiasm  if  it 
could  be  accomplished  without  a  corresponding  in- 


crease in  the  length  of  the  tests.  Thjs  paper  is 
addressed  to  the  problem  of  showing  precisely  how 
some  new  developments  jn  statistrcal  theory  make 
this  goal  attainable.  More  specifically,  the  present 
paper  indicates  precisely  how  these  Bayesian 
methods  could  be  integrated  into  an  ongoing  IPI 
program.  In  order  to  lay  a  proper  foundation,  one 
describing  the  exact  nature  of  the  measurement 
problem  In  IPI,  we  propose  to  confine  discussion  to 
One  major  component  of  the  system,  the  math- 
ematics program.  To  this  end,  a  general  de^ 
scription  of  the  assessment  instruments  used  >n  IPI 
mathematics  is  contained  in  the  next  section. 

The  mathematical  and  statistical  models  which 
form  the  basis  of  the  proposed  application,  and  the 
outline  of  this  application,  are  based  on  the  work  of 
Novick,  Lewis,  and  Jackson  (1973),  and  the  ampli- 
fications contained  in  Lewis.  Wang,  and  Novick 
(1973).  Wang  (1973),  and  Wang  and  Lewis  (1973a. 
1973b).  A  theoretical  discussion  of  these  methods  is 
contained  in  Hambleton  and  Novick  (1973).  The 
Bayesian  methods  of  statistical  inference  developed 
in  these  papers  combine  direct  observation 
information  on  each  student  with  certain 
background  information,  to  permit  more  accurate 
decision-making  than  would  be  possible  without  the 
use  of  this  background  information.  The  use  of  this 
background  information  makes  possible  the  gain  in 
accur^^cy  without  additional  testing. 


THE  IPI  MATHEMATICS  PROGRAM 


Ferguson  (1970a)  provides  a  detailed  description 
of  the  IPI  Mathematics  program.  Highlights  of  that 
description  are  provided  in  subsequent  parts  of  this 
section  In  particular,  attention  is  given  both  to  the 
structure  of  the  curriculum  and  to  the  test  model 
which  plays  such  an  important  role  in  the 
management  of  the  program. 

The  Curriculum 

Figure  1  conveys  the  general  organization  of  the 
mathematics  curriculum.  Ten  content  areas, 
Numeration/Place  Value,  Aodltion/Subtraction. 
Multiplication,  Division,  etc.,  are  identilied;  each 
occurring  at  various  levels  of  difficulty.  The  ten 
areas  are  listed  in  a  hier^^rchical  order  that  Is 
followed  in  instruction  The  intersection  ol  each 
level  with  a  specific  content  a  ea  determines  a  unit 
that  consists  of  a  set  of  tehaviorally  defined 
objectives  or  sfr/Z/s.  Each  nimber  in  the  table 


indicates  the  number  of  skills  in  the  unit.  Thus,  E 
level-Systems  of  Measurement  is  a  unit  that  consists 
of  a  set  of  five  behavioral  objectives  (skills)  which 
share  a  similar  content  but  are  less  difficult  than  the 
skills  contained  in  the  F  (evel-Systems  of  Mea- 
surement.  The  absence  of  a  number  at  any  position 
in  the  chart  indicates  that  no  unit  exists  for  the 
corresponding  content  area  and  level.  Atthe  bottom 
of  Figure  1,  we  have  listed  the  specific  behavioral 
objectives  for  E  level-Systems  of  Measurement. 

The  Test  Model 

As  previously  indicated,  the  assessment  instru- 
ments in  IPI  pcrfortn  a  dual  role  in  the  program, 
serving  both  a  placement  and  a  diagnostic  function. 
The  tests  cire  placement  oriented  in  the  sense  that 
they  locate  a  students  position  in  the  curriculum 
with  respect  to  the  skills  forwhich  he  lacks  sufficient 
proficiency,  but  for  which  he  has  the  necessary 
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Level 


A 

B 

c 

n 

F 

F 

G 

— H  ^  ^ 

Numeration/Place  Value 

15 

9 

14 

5 

6 

7 

6 

Addition/Subtraction 

17 

12 

13 

10 

4 

4 

6 

Multiplication 

4 

7 

9 

7 

4 

3 

Division 

3 

4 

7 

9 

5 

6 

Fractions 

3 

3 

6 

7 

11 

8 

8 

Money 

1 

1 

5 

5 

Time 

6 

6 

4 

4 

2 

Systems  of  Measurement 

3 

6 

6 

5 

5 

6 

Geometry 

3 

2 

4 

6 

4 

2 

Applications 

3 

8 

9 

5 

4 

6 

Behavioral  Objectives 

£  Level-Systems  of  Measurement 

1.  Givenaruler,  the  student  measures  a  line  segment  with  theindicated  degree  of  precision  LIMIT  smaflestunit 
of  precision  1/8  inch;  line  segments  to  10  inches. 

2.  Given  20  cut-out  regions  that  are  each  1 -inch  squares  and  an  illustration  of  a  rectangular  region.the  student 
uses  the  1-inch  squares  to  determine  the  area  of  the  given  rectangular  region  LIMJT  areas  <  20  square 
inches.  Length  of  sides  of  rectangles  must  be  multiples  of  1  inch. 

3.  Giventhemeasuresof  the  sides  of  a  rectangular  region,  the  student  determinesthe  area  of  that  region  LIMIT" 
integral  measures,  one  unit  of  measure  per  problem;  units  of  measure— Inches,  feet,  yards,  miles 

4.  Given  the  measure  of  the  sides  of  a  rectangular  region,  thestudent  determines  the  perimeter  and  the  area  of 
that  region.  LIMIT.  At  least  one  of  the  measures  (length,  width)  must  be  integral;  both  measures  must 
be  <  100;  one  measure  may  be  a  common  fraction  <  ^  with  denominator  <  10;  1  unit  ofmeasure  per  problem; 
units  of  measure— inches*  feel,  yards,  miles. 

5.  Given  a  weight  measurement,  the  student  completes  a  statement  to  show  an  equivalent  measurement  m  a 
different  unit  of  weight  measure.  Given  a  word  problem  that  requires  conversion  of  a  gi^'en  weight 
measurement  expressed  in  standard  units  to  an  equivalent  weight  expressed  in  another  standard  unit,  the 
student  solves  the  problem  and  writes  theansv\fer  with  the  appropriate  label  LIMIT  units-bounces,  pounds, 
tons. 


Fig.  1.  Matrix  of  Units  in  the  IPI  Mathematics  Curriculum. 


prerequisite  skills  so  that  he  can  begin  work.  The 
same  tests  are  diagnostic  In  that  they  provide 
information  that  identifies  skills  in  which  the  student 
has  not  achieved  sufficient  proficiency  and  also 
provide  insight  as  to  specific  facets  of  these  skills  on 
which  instruction  is  required.  A  review  of  the  variou?* 
tests  utilized  in  the  mathematics  program  follows. 

Curriculum  PiaCBmsnt  Tssts 

Upon  entrance  to  the  mathematics  program,  the 
placement  tests  provide  a  global  picture  of  each 
student  regarding  his  level  of  proficiency  with 
respect  to  the  skills  in  each  unit  of  the  curriculum. 
The  data  generated  by  the  placement  tests  are  used 
to  develop  a  profile  for  each  student  indicating  those 
units  in  which  he  has  sufficient  proficiency  in  all  of 
the  skills  and  those  in  which  he  has  insufficient 
proficiency  For  example*  the  outcome  of  a 
placement  test  might  yield  a  profile  indicating 
sufficient  proficiency  in  all  of  the  skills  in  level  D  of 
the  curriculum,  and  insufficient  proficiency  in  the 
skills  of  units  at  a  higher  level  of  difficulty.  In  this 
casei  the  student  would  begin  work  in  unitsat  level  E 
of  the  curriculum  More  typically,  a  student  might 
demonstrate  proficiency  at  level  D-Numer- 
ation/Place  Value,  level  F-Addition/Subtraction, 
level  E-Money.  level  C-Time.  and  perhaps  level  D  in 
all  other  areas  Such  astudent  would  probably  then 
begin  instruction  in  level  C-Time,  this  being  the 
lowest  level  in  the  area  hierarchy  at  which 
instruction  is  prescribed. 

Because  of  the  global  nature  of  placement  tests, 
they  must  assess  a  very  large  domain  of  math- 
ematics skills  Consequently,  practicality  demands 
that  the  tests  include  only  a  small  number  of  items 
on  key  objectives  in  each  unit  of  the  curriculum. 
Thus,  important  placement  decisions  are 
necessarily  dependent  on  tests  with  a  small  number 
of  items. 

Unit  PretQsts 

Once  a  placement  test  has  been  used  to  determine 
a  profile  for  a  student,  a  decision  can  be  made,  as 
indicated  in  the  previous  section,  regarding  the  unit 
on  which  the  student  begins  his  work.  At  this  point,  a 
unit  pretest  is  administered  to  identify  the  specific 
objectives  in  the  unit  for  which  the  student  has 
sufficient  {Insufficient)  proficiency.  Each  pretest 
consists  of  several  short  subtests,  one  for  each 
objective  in  the  unit. 

It  is  possible  for  a  student  to  demonstrate 
sufficient  competency  on  all  objectives  in  the  unit.  11 
this  were  to  occur,  the  student  would  continue 


working  at  the  same  level,  but  proceed  to  the  next 
unit  in  the  area  hierarchy  where  ne  would  be  given 
another  unit  pretest.  Thus,  the  pretest  provides 
additional  information  about  a  student,  information 
which  is  focused  at  the  level  determined  by  the 
placement  test. 

The  pretest  decision  can  and  sometimes  does 
override  a  part  of  the  placement  decision.  This 
occurs  when  proficiency  is  demonstrated  by  the 
student  in  areas  and  at  levels  not  indicated  by  the 
placement  test.  Thus,  the  JPI  testing  paradigm 
initially involvesatwostage  semisequenttal  testing 
programwiththeplacementtest  largely  determining 
the  level  at  which  more  intensive  testing  is  to  take 
place. 

After  the  unit  pretest  has  identified  the  specific 
skills  for  which  the  student  requires  instruction, 
student  test  performance  on  each  of  these 
objectives  is  examined  by  the  teacher  to  identify 
particular  typesof  errors  Or  patterns  oferrors.lnthis 
manner  learning  materials  and/or  experiences 
consonant  with  the  individuals  needs  can  be 
prescribed. 

The  typical  pretest  includes  between  six  and 
(preferably)  ten  items  for  each  obiective.Obviousty, 
the  size  of  the  domain  of  items  varies  with  the 
particular  skill.  Usually,  tiowever,  the  domain  is 
quite  large.  Thus,  important  instructional  decisions 
are  often  b£©ed  on  student  performance  on  a  small 
number  of  items  that  have  been  represer^tatlvely 
sampled  from  a  very  large  domain.  The  relative 
shortness  of  the  tests  can  certainly  bejustifiedfrom 
a  practical  point  of  view.  Longer  tests  might  be 
considered  repressive  and  would  certainly  exceed 
reasonable  bounds  m  terms  of  the  proportion  of  time 
given  over  to  them  within  the  total  instructional 
process.  Thus,  it  would  appear  that  the  key  to  more 
effective  and  more  reliable  decisions  ttes  not  in 
increasing  the  length  of  the  tests  beyond^  sayt  ejght 
or  ten  itemst  but  rather  in  making  better  use  of  the 
data  available  within  the  present  system. 

Curriculum  EmbBddsd  Te^ts 

These  short  "quizzes"  measure  the  student's  level 
of  proficiency  in  a  single  skill  within  the  curhculum. 
The  written  instructional  material  (or  each  skill  in  a 
mathematics  unit  contains  two  curnCLlum  em- 
bedded tests  (CETs).  The  tests  are  self-evaluatron 
devices  used  by  the  student  as  a  check  on  his 
progress  as  it  relates  to  his  work  on  a  given  skill. 
Thus,  the  student  who  has  completed  several 
learning  activities  related  to  the  development  of  his 
proficiency  in  a  particular  skill  might  take  a  GET  to 
determine  whether  he  has  attained  sufficrent 
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proficiency  at  this  point  or  whether  he  needs  to 
complete  additional  steps  m  the  instructional 
process. 

The  GET  typically  consists  of  from  four  to  six 
items.  Because  these  short  tests  serve  primarily  as 
self-checks  for  the  student,  and  because  no  crucial 
instructional  decision  ts  dependent  upon  student 
performance  on  these  tests*  they  seem  to  be  ade- 
quate for  the  task  which  they  serve. 


Unit  Po$tte$t$ 

These  instruments  are  equivalentformsoMheunit 
pretests.  They  are  generally  administered  after  the 
student  has  concluded  learning  activities  for  all 
skills  for  which  he  was  identified  as  being 


insufficiently  proficient  on  the  unit  pretest.  On  the 
basis  of  the  student  s  performance  on  the  posttest. 
he  is  either  advanced  to  the  next  unit  or  required  to 
work  with  additional  instructional  materials  on  those 
skills  for  which  his  test  performance  did  not  indicate 
that  he  achieved  a  sufficient  level  of  proficiency  A 
student  generally  does  not  advance  to  a  new  unit 
until  he  has  demonstrated  sufficient  proficiency  for 
all  objectives  of  the  current  unit. 

As  with  the  pretests^  decisions  resulting  from  an 
analysis  of  posttest  data  rely  upon  tests  which  gen- 
erally contain  a  small  n  umber  of  items.  Because  in- 
correct proficiency  decisions  can  be  detrimental  to 
the  student's  progress,  a  procedure  which  could  add 
Substantially  to  the  accuracy  of  the  decision  with- 
out increasing  the  length  of  the  test  would  be  most 
worthwhile. 


THE  INSTRUCTIONAL  DECISION  PROCESS 


In  this  section,  the  process  by  which  test  data  are 
used  to  make  instructional  decisions  is  briefly 
summarized.  In  addition,  a  discussion  of  the  nature 
and  consequences  of  decision  errors  resulting  from 
the  analysis  of  test  data  is  presented. 

A  Summary  of  ths  Dsctston  Process 

Gross  placement  tests  which  sample  a  broad 
cross  section  of  the  important  skills  in  each  unit  of 
the  mathematics  curriculum  are  administered  upon 
each  student's  entry  into  the  IPI  program.  Scoredata 
resulting  from  these  tests  are  used  to  determine  a 
pruftle  suggesting  the  student's  level  of  proficiency 
in  each  content  area  of  the  curriculum. 

At  this  point,  the  student  completes  a  pretest  for 
the  first  unit  in  the  curriculum  continuum  in  whict) 
his  level  of  proficiency  is  insufficient.  The  profile 
resulting  from  the  pretest  identifies  those  skills  for 
which  learning  materials  and/or  experiences  are 
required  if  the  student  is  to  achieve  the  specified 
level  of  performance.  During  the  instructional 
process^  curnculum  embedded  tests  are  available  to 
the  student  as  a  means  of  self-evaluation  and  an 
estimate  of  progress  as  he  works  on  the  skills.  After 
he  has  completed  work  on  all  skills  in  the  unit  and  is 
satisfied  that  he  has  sufficient  competency  in  all  of 
the  unit  skills,  he  is  administered  a  posttest  which 
verifies  his  progress  or  identifies  those  skills  for 
which  additional  instruction  is  indicated-  Once  the 
unit  IS  successfully  completed,  the  student  ad- 
vances to  the  next  unit  on  his  prescription  where  he 
IS  administered  a  pretest  and  the  cycle  is  repeated, 


ThB  Nature  and  Effect  of  Decision  Errors 

The  placement  tests,  pretests,  and  posttests  are 
used  primarily  to  verify  that  a  student  either  has 
sufficient  proficiency,  i.e.  mastery,  in  a  given  set  of 
skills  or  that  he  has  an  inadequate  level  of  pro- 
ficiency in  those  skills.  Clearly,  it  isdeciidblethatthe 
mastery  decisions  for  a  student  be  as  accurate  as 
possible.  The  importanceof  accuracy  of  the  mastery 
decision  for  a  student  is  perhaps  best  emphasized 
by  a  discussion  of  the  consequences  of  an  incorrect 
decision. 

As  previously  indicated,  the  IPI  tests  are  con- 
structed by  sampling  items  j  rem  the  domain  of  items 
for  the  objectives  included  on  the  tests.  Since  any 
sampling  which  does  not  exhaust  t^  e  population  of 
items  for  an  objective  can  lead  H  an  incorrect 
mastery  decision  and  since  exhaustive  testing  is 
impossible,  it  is  necessary  to  tolerate  the  risk  of 
making  wrong  decisions.  In  an  IPI  context,  a  Type  I 
( a  )  error  occurs  when  an  examinee  has  sufficient 
proficiency  in  a  skill  but  the  outcome  of  the  testing 
suggests  that  he  does  not.  As  a  result,  he  is 
prescribed  work  lessons  which  may  serve  no  sig- 
nificant function.  A  Type  It  <  /J )  error  occurs  when- 
ever the  examinee,  in  fact,  lacks  proficiency  in  a  skill 
but  on  the  basis  of  test  results  is  said  to  have 
sufficient  proficiency.  The  consequence  of  a  Type  1 1 
error  is  that  needed  remedial  instruction  is  not 
provided.  A  T/pe  II  error  is  perceived  to  be  poten- 
tially more  senous  than  aTypel  errorsincetheType 
II  error  could  easily  result  in  the  student  having 
difficulty  proceeding  through  a  unit  and  might 
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eventually  lead  to  an  impasse  in  instruction; 
whereas,  the  Type  1  error  wil!  at  worst  require  that 
the  student  pursue  a  review*like  study  of  skills  in 
which  he  is  already  proficient. 

Although  it  is  clear  that  the  magnitude  of  the 
consequences  of  an  incorrect  proficiency  decision 
for  a  student  varfes  with  the  direction  of  the  error,  it 
is  equally  clear  that  in  both  cases  the  error  may  have 
detrimental  effects  for  the  student.  The  fact  that  the 
tests  on  which  these  decisions  are  based  have  a 
small  number  of  items  per  skill  suggests  that  such 
errors  probably  occur  quite  frequently.  Given  the 
constraints  imposed  by  a  program  which  already 
has  a  heavy  testing  component,  increasing  the 
length  of  the  tests  is  not  a  tractable  method  for 
achieving  increased  accuracy  in  the  mastery 
decision  process.  However,  it  may  very  well  be 
possible  to  incorporate  additional  information  into 
the  decision  process  and  ;  »us  improve  the  overall 
accuracy  of  the  decisions  being  made.  It  is  this 
hypothesis  to  which  the  remainder  of  this  paper  is 
addressed. 

In  IPL  as  in  all  individualized  instructional 
programs,  decisions  are  focused  around  the 
individual  student.  If  a  statistical  procedure  that 
uses  information  other  than  that  contained  in  the 
immediate  direct  observations  on  the  student  \^ 
contemplated,  then  a  Bayesian  procedure  incor- 
porating prior  information  on  each  student  comes  to 
mini  This  information  would  consist  of  results  of 
the  student's  performance  on  previous  instructional 
units.  In  this  way.  interindividual  variability  on  prior 
test  performance  would  be  helpful  in  making  current 
decisions. 

The  problem  with  this  thinking  is  that  the  entire 
thrust  of  individualized  instructicn  works  toward  a 
reduction  of  iilterstjdent  variability  of  test  results.  A 
student  moves  ahead  to  a  new  unit  of  instruction 
only  when,  it  is  thought,  he  is  prepared  to  do  so. 
Indeed,  he  is  encouraged  not  to  take  the  unit 
posttest  until  there  is  strong  evidence  that  he  is 
prepared  to  perform  well  on  it.  A  great  deal  of 
posttest  score  variability  is  in  fact  observed,  but 
much  of  it,  though  not  all,  results  from  unreliability 
due  to  the  necessarily  short  length  of  these  tests. 
Thus,  realistically,  there  is  little  or  no  useful 
differential  prior  information  about  the  individual 
student. 

On  the  other  hand,  there  is  a  great  deal  of 
information  available  abOkit  the  instructional 
program  Quite  specific  information  is  available 
concerning  the  distribution  of  the  percentage  of 
items  answered  correctly  by  students  (Novjck, 
Lewis,  and  Jackson.  1973).  and  it  is  thus  possible  to 
make  infeiences  about  the  true  level  of  functioning 


of  each  student,  and  the  mean  and  standard 
deviation  of  these  true  values  in  the  population  of 
students.  Of  course,  if  the  instructional  programs 
were  comploioiy  efficient  and  the  students  were 
without  human  frailties,  there  would  be /70varratiOn 
in  true  levels  of  functioning  of  students  on  posttests. 
A  student  would  remain  in  a  unit  only  until  that 
instant  at  which  his  level  of  functioning  attained  the 
prespecified  criterion.  Nothing  approaching  this  is 
possible  with  present  instructional  technology, 
(-lowever.  If  we  knew  this  were  the  true  state  of 
affairs,  then  we  would  ignore  individual  test  scores 
and  use  our  information  on  the  group  mean  and 
variance  to  make  a  positive  proficiency  decision  for 
all  students. 

In  the  real  world  of  Individually  Prescribed 
Instruction  there  will  be  some  variation  in  true  levels 
of  functioning  among  students  on  posttests.  The 
delicate  iTi^iinner  in  which  background  information  is 
combined  with  the  direct  observational  data  in  the 
Bayesian  decision  process,  and  the  increment  in 
decision-making  accuracy  resulting  therefrom  is 
detailed  in  Novick^  Lewis^  and  Jackson  (1973)  and 
Lewis,  Wang,  and  Novick  (1973). 

Finally,  we  may  note  one  additional  source  of 
background  information  that  can  be  utilized  when, 
as  in  IPI,  testing  involves  joint  measurement  on 
several  skills,  simultaneously.  !n  this  situation  and 
assuming  some  relationship  among  the  skills,  it  is 
possible  to  use  the  collateral  information  contained 
in  the  t  -  1  of  t  tests  scores  for  each  person  to  help 
estimate  each  t-th  test  score.  Thus,  if  a  person 
scoredhighlyint  -  1  subtests  and  a  little  less  highly 
in  the  t-th,  we  would  suspect  that  this  might  be  due/f? 
part  to  bad  luck  or  carelessness,  &nd  we  would  be 
inclined  to  make  some  adjustment  in  our  estimate  of 
his  proficiency  on  that  skill  The  Bayesran  theory 
and  methods  described  by  V.'angand  Lewis  (1973a. 
1973b)  provide  the  rationale  and  prescription  for 
doing  this. 


impiementBtion  Procedures 

The  decision  analysis  procedures  employed  by 
teachers  and  students  in  the  IPI  program  must  not 
be  overly  coit^plex.  Thus,  the  f  mal  output  of  the  data 
analysis  procedures  used  to  judge  the  leve!  of 
proficiency  of  a  student  must  be  so  simple  that 
teachers,  aides,  and  even  students  can  read  the 
results,  interpret  them,  and  then  lake  whatever 
action  is  indicated.  It  will  be  permissible  to  use 
sophisticated  statistical  methods,  but  teachers, 
aides,  a.-;d  students  must  not  be  required  to 
understand  much  more  than  is  contained  m  this 
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paper.  In  short,  although  \\  is  not  necessary  that 
teachers  and  students  understand  the  details  of  the 
analysis,  they  must  be  provided  information  which 
facilitates  theirinstructionaldecisjon  making.  Inthe 
following  section,  procedures  for  dealing  with  the 
preceding  concerns  are  discussed. 

The  coUection  and  anaiysts  of  data.  During  the 
past  sevBral  years,  considerable  investigation  has 
been  underway  into  the  feasibility  of  using  a 
computer  as  an  integral  part  of  the  IPI  program.  A 
thorough  discussion  of  the  most  recent  devel- 
opments is  available  in  a  progress  report  (Block. 
Carlson.  Fitzhugh.  et  al..  1973)  recently  released  by 
the  Learning  Research  and  Development  Center  at 
the  University  of  Pittsburgh.  Earlier  reports  include 
Cooley  and  Glaser  (1969).  Ferguson  (1970b.  1971). 
and  Ferguson  and  Hsu  (1971).  The  activities 
described  in  these  reports  emphasize  somewhat 
visionary  ideas  for  how  the  computer  can  best  be 
employed  in  an  Individualized  program  of 
instruction.Althoughthesestudiesinclude  the  more 
conventional  modes  of  computer-assisted 
instruction,  they  extend  far  beyond  into  such  areas 
as  computer  testing  and  instructional  management. 

It  is  in  this  latter  area,  instructional  management, 
that  Bayesian  procedures  for  determining  pro- 
ficiency decisions  would  best  seem  to  reside.  Work 
in  this  area  has  been  concerned  with  how  the 
computer  can  assist  in  the  planning  and  subseq  uent 
monitoring  of  both  short-  and  long-term  instruction 
for  individual  students.  Thus,  it  wouki  seem 
appropnate  to  incorporate  a  decision-making  pro- 
cedure concerned  with  individual  proficiency  level 
in  some  skill,  or  set  of  skills,  as  an  element  of  the 
instructional  management  component  of  the  IPI 
program.  Specifically,  the  computer  might  be  used 
lo  receive  test  data  on  a  student  and  combine  this 
with  previously  acquired  information  on  other 
students  in  this  IPI  program,  analyze  the  data  using 
Bayesian  analysis  techniques,  and  then  print  out  a 
report  indicating  the  confidence  which  one  could 
place  in  deciding  that  the  student  is  proficient  in  a 
given  skill  at  some  prespecified  level  of 
performance.  A  more  detailed  discussion  of  how  this 
procedure  might  work  is  nowprovided  in  the  context 
of  IPI  posttests.  Procedures  similar  to  those 
described  below  would  apply  for  placement  te^ts 
and  pretests  as  well 

Deveiopment  and  use  oi  a  postte^t  profile.  The 
primary  purpose  fof  administering  a  placement  test, 
a  pretest,  or  a  posttest  is  to  acquire  data  which  can 
be  used  to  evaluate  a  students  instructional  needs. 
When  a  student  is  administered  a  posttest,  he  is 
presumed  to  have  had  instruction  in  those  skills  for 
which  he  lacked  sufficient  proficiency  at  the  timehe 
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was  administered  the  unit  pretest.  The  posttest 
either  affirms  the  students  success  in  acquiring  the 
skills  or  calls  attention  to  those  skills  in  which 
additional  work  is  required  before  hecanproceedto 
the  next  unit.  Thus,  the  only  information  which  the 
teacher  and  student  need  is  a  simple  statement 
regarding  the  level  of  proficiency  at  which  the 
student  has  performed  on  each  skill  in  the  unit. 
Figure  2  shows  an  IPI  posttest  profile  based  onatest 
consisting  of  five,  eight-item  subtests,  each 
measuring  proficiency  level  on  a  particular  skill. 


Level  E-Mult[pl[cation/Division 


Skill 

Percent  Correct 

1  ■ 

87.5 

2 

87.5 

3 

75.0 

4 

100.0 

5 

67.5 

Rg.  2.  Sample  of  Posttest  Profile  Currently  in  Use 
in  IPI. 


Presently,  the  posttest  profile  names  each  skill  in 
the  unit  and  lists  the  pexcBntage  of  items  which  the 
student  answered  correctly.  Given  the  sample 
profile  in  Figure  2  and  a  criterion  (cutoff)  score  of 
85%.  it  is  likely  that  the  studentwould  be  called  upon 
to  undertake  additional  work  in  the  3rd  and  5th  skills 
of  the  unit. 

Under  the  proposed  change,  rather  than 
evaluating  student  proficiency  solely  on  the  posttest 
results,  additional  data  would  be  incorporated 
within  the  decision  analysis  process,  and 
furthermore,  the  quantity  reported  would  be  an 
index  relating  the  student's  estimated  proficiency  to 
a  stipulated  standard.  However,  it  should  be 
emphasized  that  although  the  nature  of  the  data 
reported  in  the  student  profile  would  change,  the 
procedures  employed  by  the  teacherand/orstudent 
to  judge  proficiency  would  remain  ^he  same 
Specifically,  the  posttest  profile,  which  presently 
contains  a  statement  of  the  percentage  of  items 
correctly  answeredfor each  skill,  would  be  alteredto 
report  the  probability  that  the  student  has  achieved 
some  prespecified  level  of  proficiency  in  each 
objective.  As  far  as  the  teacher  or  student  Is 
concerned,  the  proficiency  decision  process  is 
exactly  the  same— judgments  are  based  on  the 
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evaluation  of  a  smgle  number  or  jndex"  for  each 
skill  Figures  provides  an  example  of  such  a  profile. 


Level  E-Systems  of  Mei^^jrement 


Skill 

Mastery  Index 

1 

80 

2 

90 

3 

76 

4 

92 

5 

40 

Ftg.  3.  Proposed  Sample  Posttest  Profile  Using 
Bayesian  Decision  Analysis  Procedures. 


data  wouid  be  m  a  ftie  permanently  maintained  on 
the  computer  and  periodically  updated.  Tl^is 
function  could  be  perlormed  automatically  by  the 
computer. 

Since  it  is  very  likely  that  many  schools  using  IPI 
will  not  have  ready  on-line  access  to  a  computer,  an 
alternative  procedure  for  providing  the  same 
decision  analysis  would  call  for  the  construction  of 
simple  ^'Mastery  Index"  tables.  These  tables  would 
permit  the  teacher,  the  aide,  or  a  student  to 
determine  the  probability  that  the  student  has 
sufficient  proficiency  in  a  skill  by  simply  entering  the 
table  with  the  number  of  items  answered  correctly 
on  each  skill  of  the  posttest.  Figure  4  serves  as  an 
example  of  such  a  table. 


Level  E-Systems  of  Measurement 


In  Figure  3,  the  column  labeled  Mastery  Index 
actually  represents  a  probability  statement.  If.  lor 
example,  the  criterion  or  cutoff  score  for  sufficient 
proficiency  is  .85,  the  Mastery  Index  column  gives 
tl^e  probat^ility  that  tl^e  student's  level  of  proficiency 
is  above  .85  for  each  skill.  In  this  case,  the  mastery 
index  for  skill  1  is  ,80.  We  see  that  the  actual  test 
performance  wasonly75%.  This  might  suggest,  very 
roughly,  a  probability  of  .50.  a  50/50  chance,  for  the 
true  level  of  functioning  being  above  .75.  However, 
the  Bayesian  analysis,  using  the  collateral 
information  has  raised  to  .80  the  probability  that  the 
student's  level  of  functioning  is  above  .85, 
Therefore,  if  v^e  would  want  to  move  a  student  on  if 
the  odds  were  better  than  three  to  one  in  favor  ol  his 
actually  being  proficient,  we  would  advance  this 
student  since  his  probability  of  mastery  is  greater 
than  .67. 

tmpfBrnsntation  mode.  A  profile  similar  to  the  one 
described  m  Figure  3  could  be  provided  in  at  leasi 
two  ways.  One  method  of  delivery  would  require  the 
availability  of  tests  which  are  administered  by 
computer  Presently,  test  adminisfi^tion  by 
computer  is  very  much  a  partof  thefeasi.')ility  study 
underway  in  IPI  Given  the  existence  of  a  unit 
posttest  on  some  specified  unit,  it  would  seem  qujte 
possible  for  sample  dd^a  generated  by  thta  computer 
test  to  be  merged  witn  a  file  containing  collateral 
data  on  student  success  m  the  system.  For  example, 
the  computer  te^  I  program  could  be  designed,  upon 
student  completion  of  the  test,  to  call  a  subroutine 
which  would  access  the  collateral  data  file,  combine 
the  two  sets  of  information,  compute  the  mastery 
mdices  (apostenon  probabilitres),  and  print  out  a 
profile  similar  to  Figure  3.  In  this  case,  the  collateral 
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Skill  1 

Number  of  items 

answered  correctly  Mastery  Index 


8 

.98 

7 

.93 

6 

.85 

5 

.73 

4 

.60 

3 

.34 

2 

.27 

1 

.12 

0 

.03 

Fig.  4.  Sampleof  Proposed  "  Mastery  Index"  Table 
for  IPI. 

Given  knowledge  of  the  number  of  items  which 
the  student  answered  correctly  out  of  a  possible 
eight  on  skill  1  of  the  level  E  posttest  for  Systems  of 
Measurement,  the  teacher  or  student  wouid  enter 
the  Mastery  Index"  table  with  that  number.  For  ex- 
ample, if  thestudent responded  correctly toseven of 
eight  items,  he  wOuJd  enter  the  table  the  left  hand 
column  v/ith  the  number  seven  and  consequently 
determine  ihat  the  probability  that  the  student  has 
the  prespecified  level  of  proficiency,  say  .85.  is  93. 
The  decision  as  to  wnether  to  move  a  student 
forward  or  not  would  depend  on  this  probability  and 
the  relative  disutilities  associated  with  the  two  kinds 
of  errors.  The  simple  methods  for  accomplishing 
this  are  described  by  Davjs.  Hickman,  and  Novick 
(1973). 


The  mdiceb  *eported  tn  thw  tables  would  have 
been  generated  at  some  earlier  time  and  would  have 
included  consideration  of  relevant  prior  data 
reaarding  student  success  on  the  skills  contained  m 
the  unit  The  tables  would  be  updated  on  a  regular 


b&^i^  *ib  ificrtfasud  nujiibers  ul  student:>  proceeded 
thfough  the  system,  thus  making  more  prioi 
information  available.  Such  an  updating  might 
occur  once  or  tvice  a  year. 


SUMMARY 


Individualized  (earning  programs  like  IPI  generate 
substantial  amounts  of  data  related  to  student 
success  on  skills  in  the  system.  Given  these  data,  it 
seems  reasonable  to  suggest  that  they  should  be 
used  to  improve  the  quality  of  instructional  decision 
making.  In  particular,  prior  data  should  be  combined 
With  sample  test  data  to  form  a  more  complete 
information  base  on  which  to  evaluate  student 
proficiency.  By  usjng  such  data  jointly,  instructional 
decisions  regarding  a  student  s  ne^ds  as  they  relate 
to  a  given  skill  or  set  of  skills  will  be  deserving  of 
more  confidence  than  present  decisions  which  are 
Curie ntly  based  soiely  on  the  student  s  performance 
on  a  short  test. 


Two  procedures  for  implementing  such  a  plan 
have  been  proposed.  One  calls  tor  the  marriage  of 
the  Bayesian  decision  analysis  procedures  with 
computer  administered  tests,  whereas,  the  other 
would  rely  on  the  teacher  or  student  to  consult  a 
t^ible  to  translate  student  test  perfoimance  to  a 
Proficiency  Index"  which  would  incorporate  both 
the  test  data  and  prior  data  regarding  student 
success  in  the  system.  The  ultimate  criterion  for 
success  of  such  3  plan  is  the  extent  to  which  it  leads 
to  improvements  in  the  instructional  decision 
process.  To  this  end,  the  next  step  is  to  implement 
the  procedures  and  evaluate  their  impact  on 
students  withm  IPI. 
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I.  Introduction 


,U1  Background 

While  the  Idea  of  developing  instructional  programs  in  our  schools 
to  meet  Individual  student  needs  Is  not  a  new  theme  in  American  education 
(see»  for  example^  Washburne^  1922;  and  Wilhelms^  1962) »  It  has  only 
been  In  the  last  decade  that  such  programs  have  been  Implemented  on  any 
large'-scale  basis  In  the  schools. 

The  basic  argument  In  favor  of  individualizing  instruction  comes  from 
a  multitude  of  research  studies  that  suggest  that  students  differ  In 
Interests^  motivation^  learning  rate»  goals^  and  capacity  for  learning 
among  other  things;  and>  therefore^  grouped-based  Instruction  on  a  common 
curriculum  is  Inappropriate  to  meet  their  educational  needs.    That  change 
In  our  schooAs  Is  obvious  when  one  notes  that  schools  provide  successful 
learning  experiences  for  only  about  oni^-third  of  our  students  (Blocks  1971). 


The  research  reported  herein  was  performed  pursuant  to  Grant  No. 
OEG-0-72-*0711  with  the  Office  of  Education,  U.S.  Department  of  Health, 
Education,  and"  Welfare,  Melvln  R.  Hovlck,  Principal  Investigator. 
Contractors  undertaking  such  projects  under  Govemment  sponsorship  are 
encouraged  to  express  freely  their  professional  judgment  In  the  conduct 
of  the  project.    Points  of  view  or  opinions  stated  do  not,  therefore, 
necessarily  represent  official  Office  of  Education  position  or  policy. 

2 

The  author  would  like  to  acknowledge  the  Insightful  comments  find 
constructive  criticisms  of  Melvln  H.  Novlck  of  The  American  College 
Testing  Program  on  earlier  drafts  of  the  manuscripts    In  addition, 
Richard  Ferguson  and  Roy  Williams  provided  many  useful  suggestions. 
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On  the  basis  of  Project  TALENT  data>  Flanagan^  et  al. ,  (1964)  reported 
that  our  current  Instructional  programs  are  Inadequate  to  handle  the 
Ic.rge  Individual  differences  In  any  age  or  grade  group*    In  addition^ 
schools  generally  fall  to  help  the  student  develop  a  sense  of 
responsibility  for  his  educational^  personal,  and  social  development 
or  to  make  realistic  educational  decisions  and  choices  about  his  future^ 
This  trend  toward  Individualization  of  Instruction  In  education  has 
resulted  In  the  development  of  a  diverse  collection  of  attractive  alternative 
models  (see,  for  example,  Gibbons,  1970;  and  Heathers,  1972)  that,  according 
to  their  supporters,  offer  new  approaches  to  student  learning  which  <  . 
provide  almost  all  students  with  rewarding  school  experiences.  These 
Include;    Individually  Prescribed  Instruction  (IPI)  (Glaser,  1968,  1970), 
Program  for  Learning  In  Accordance  vlth  Needs  (PLAN)  (Flanagan,  1967, 
1969),  Computer-Assisted  Instruction  (CAI)  (Suppes,  1966;  Atkinson,  1968; 
Atkinson  and  Wilson,  1969),  Individualized  Mathematics  Curxlculum  Project 
(De  Vault:,  Krlewall,  Buchanan,  and  Quilling,  1969),  and  Mastery  Learning 
(Carroll,  1963,  197f^,  Bloom,  1968;  and  Block,  1971)*    All  of  the  models, 
as  well  as  many  others,  represent  significant  steps  forward  In  Improving 
learning  by  Individualising  Instruction*    They  strive  to  actively  Involve 
the  student  In  the  learning  process,  allow  students  In  the  same  class 
to  be  at  different  points  In  the  curriculum,  and  permit  the  teacher  to 
give  more  Individual  attention. 

In  important  aspects  of  these  Individualized  Instructional  programs 
such  as  the  construction  of  instructional  materials  (Popham,  1969j 
Smith,  1969),  curriculum  design  (Wlttrock  and  Wiley,  1970)  and  computer 
riianagement  (Baker,  1971;  Cooley  and  Glaser,  1969),  there  are  substmtial 
bodies  o£  knowledge.    It  is  perhaps  surprising  to  note  then  that  the 
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amount  of  information  currently  available  on  the  testing  methods  and 
decision  procedures  for  these  programs  is  quite  limited.    It  is  this 
component  that,  in  principle^  facilitates  the  efficient  movement  of 
students  through  the  instructional  program. 

One  reason  for  a  lack  of  information  is  that  measurement  requirements 
wituin  the  context  of  many  of  the  new  programs  require  new  kinds  of  tests* 
These  are  the  criterion-referenced  tests  which  are  constructed  and 
interpreted  in  ways  quite  different  from  the  norm-r^f erenced  tests  which  are 
more  familiar  to  most  practitioners  in  the  field  (Popham  and  Husek,  1969; 
Glaser  and  Nitko^  1971;  Hambleton  and  Novlrk,  1973). 

Since  onft  of  the  major  purposes  of  * -idividualized  programs  Is  to 
maximize  the  opportunity  for  all  students  to  learn^  it  follows  that  tests 
used  to  monitor  student  progress  should  be  keyed  to  the  instruction* 
Further^  they  should  provide  information  that  can  be  used  to  measure 
progress  along  an  absolute  ability  continuum*    Norm-ref'^.renced  uests  ure 
constructed  specifically  to  facilitate  making  comparisons  among  students; 
hence>  they  are  not  very  well  suited  for  making  most  of  the  instructional 
decisions  required  in  individualized  instructional  programs. 

1.2    Criterion-Referenced  Testing  and  Measurement 

Much  of  the  discussion  in  the  arei  of  criterion^-referenced  testing 
and  measurement  (for  example,  see  Blockt  1971;  Ebel,  1971;  Glaser  and 
NitkOi  1971;  and  Hambleton  and  Novick,  1973)  stems  from  different 
understandings  as  to  the  basic  purpose  of  testing  in  the  instructional 
models  described  in  the  previous  section.    It  would  seem  that  Ln  most 
cases  the  pertinent  question  is  whetlier  or  not  the  individual  has  attained 
some  prescribed  degro.e  of  competence  on  an  instructional  performance  task* 
Questions  of  precise  achievement  levels  and  comparisons  among  individual!^ 
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on  these  levels  seem  to  be  largely  irrelevant.    In  many  of  the  new 
instructional  models^  tests  are  used  to  determine  on  which  Instructional 
objectives  an  examinee  has  met  the  acceptable  performance  level  standard 
set  by  the  model  designer.    This  test  Information  Is  usually  used 
Immediately  to  evaluate  the  student's  mastery  of  the  instructional 
objectives  covered  In  the  test>  so  as  to  appropriately  locate  hltn  for 

next  Instruction  (Glaser  and  Nltko^  1971).    Tests  especially  deeiigned 
for  this  particular  purpose  have  come  to  be  known  as  crlterlon^refcrenced 
tests.    Criterion-referenced  tests  are  specifically  designed  to  meet 
the  measurement  needs  of  the  new  Instructional  models.    In  contrast^  the 
better  known  norm-re fe rence d  tests  are  principally  designed  to  produce 
test  scores  suitable  for  ranklxig  Individuals  on  the  ability  measured  by 
the  test.    A  very  flexible  definition  of  a  criterion-referenced  test  has 
been  proposed  by  Glaser  and  Nltko  (1971):    "...[a  teat]  that  Is  deliberately 
constructed  so  as  to  yield  measurements  that  are  directly  interpretable 
In  terms  of  specified  performance  standards."    According  to  Glaser  and 
Nitko  (1971),  "The  performance  standards  are  usually  specified  by  defining 
some  domain  of  tasks  that  the  student  should  perform.  Representative 
samples  of  tasks  from  this  domain  are  organized  Into  a  test.  Measurements 
are  taken  and  are  used  to  make  a  statement  about  the  performance  of  each 
Individual  relative  to  that  domain."    Distinctions  between  norm- 
refetenced  tests  and  criterion-referenced  tests  have  been  presented  by 
Glaser  (1963) >  Glaser  and  Nltko  (1971),  Livingston  (19/2) ,  Popham  and 
Husek  (1969),  Ebel  (1971),  Block  (1971),  Hambleton  and  Gorth  (1971), 
and  Hieronymous  (1972) . 

Hambleton  and  Novick  (1973)  have  discussed  the  evaluation  ot  criterion- 
referenced  tests  In  practical  situations.  In  their  formulation,  ruHabiJUy 
takes  the  form  of  an  Index  Indicating  the  consistency  of.  decision  makLtig 
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across  parallel  forms  of  the  criterion-referenced  test  or  across  repeated 
measurements.    Validity  takes  the  same  £orm  except,  of  course,  that  a  new 
test  or  some  other  appropriate  measure  serves  as  the  criterion.  Both 
reliability  and  validity  concepts  are  reformulated  in  straightforward 
decision^theoretic  terms.    However,  at  this  stage  of  the  development  of 
a  theory  of  criterion- referenced  measurement,  the  establishment  of 
cut-off  scores  is  primarily  a  value  judgment*    [Further  clarification  is 
provided  by  Hambleton  and  Novick  (1973),  Millman  (1973),  and  Block 
(1972)*] 

1*3    Instructional  Models  Under  Consideration 

The  major  concern  in  this  paper  is  with  instructional  models  that 
include  a  specification  of  the  curriculum  in  terms  of  behavioral 
objectives,  detailed  diagnosis  o£  the  entering  competencies  of  students, 
the  availability  of  multiple  instructional  resources,  individual  pacing 
and  sequencing  of  material,  as  well  as  the  careful  monitoring  of 
student  progress* 

In  the  programs  under  consideration,  Computer^Hanaged  Instruction 
(CMI)  ^,s  an  opttional  feature.    Under  CMI  the  goal  is  for  the  computer  to 
service  classroom  terminals  which  assist  the  classroom  teacher  in 
assessing  a  student's  strengths  and  weaknesses,  and  to  prescribe 
instructional  sequences  (Cooley  and  Glaser,  1969).    Project  PLAN  and 
CAI  are  Implemented  in  a  CMI  mode  whereas  IPI  and  Mastery  Learning  are 
not » 

In  summary^t  the  goals  of  individualized  instructional  programs 
developed  along  the  general  lines  of  the  specifications  above  are  to 
enable  students  to  work  through  the  units  of  instruction  at  a  pace 
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reasonable  for  them>  to  develop  self-direction  and  self-initiation >  to 
encourage  self-evaluation  as  well  as  motivation  for  learning^  and  to 
demonstrate  mastery  in  a  variety  of  skills* 

Cronbach  (1967)  reported  on  three  major  patterns  of  dealing  with 
individual  differences  which  ppovide  a  framework  for  the  models  considered 
in  this  paper*    Patterns  of  dealing  with  individual  differences  in  the 
school  can  be  described  in  terms  of  the  extent  to  which  educational  goals 
and  instructional  methods  are  varied.    In  one  pattern^  the  educational 
goals  and  instructional  methods  are  relatively  fixed  and  inflexible. 
Individual  differences  are  handled  mainly  by  dropping  students  from  the 
program  when  they  begin  to  encounter  difficultyi    In  a  second  pattern^ 
goals  are  selected  for  students  on  the  basis  of  interest  and  potential* 

They  are  then  channeled  into  one  fixed  program  or  another.  Individual 
differences  are  handled  by  providing  multiple  optional  programs.  The 
models  we  describe  in  this  paper  fit  into  a  third  pattern  where  goals  and 
instructional  resources  are  individualized  for  the  purpose  of  maximizing 
learning  * 

1.4    Purposes  of  the  Investigation 

The  success  of  individualization  depends  to  a  considerable  extent  on 
how  effectively  teachers  and  studants  make  decisions  as  to  the  ma±!tary  of 
specific  instructional  objectives^  the  development  of  individual 
prescriptions^  the  selection  of  instructional  resources^  etc.  However^ 
various  writers  including  Baker  (1971)  and  Glaser  and  Nitko  (1971)  have 
commented  rather  critically  on  existing  testing  techniques  and  procedures* 
Relevant  background  for  improving  such  a  situation  would  certainly  include 
a  review  of  the  testing  models  of  some  of  the  more  commonly  used 
individualized  instructional  programs.    Such  a  review  would  assist  in 


92 


7 


defining  the  kinds  of  decisions  that  are  tnade^  and  the  Information  on 
which  the  decisions  are  based*    This  should  provide  a  basis  for  developing 
testing  methods  and  decision  procedures  specl£lcally  designed  for  use 
within  the  context  of  these  models.     (Although  it  would  be  Ideal  to 
develop  a  general  measurement  model  to  cover  all  the  Instructional 
models,  we  are  not  prepared  In  this  paper  to  advance  such  a  model*) 

The  first  purpose  of  the  Investigation  was  to  provide  a  description 
of  the  testing  models  that  are  currently  being  used  In  selected 
Individualized  instructional  programs*    Three  programs  wete  selected 
for  study:    Individually  Prescribed  Instruction,  Program  for  Learning  In 
Accordance  with  Needs,  and  Mastery  Learning*    [These  laodels  as  well  as 
others  are  also  discussed  by  Baker  (1971);  however,  he  was  concerned 
with  their  computer-based  Instructional  management  systems  which  are  of 
only  secondary  interest  in  this  paper*]    These  programs  were  selected 
in  this  study  because  they  are  among  the  best  known  and  because  there 
is  a  substantial  amount  of  information  available  on  each*    In  the 
following  Sections,  an  Introduction  is  provided  for  each  instructional 
model*    The  introduction  Includes  a  brief  history  of  the  program,  the 
content  areas  covered,  and  an  indication  of  the  extent  of  impleinei*tatio;u 
Also,  a  description  of  each  instructional  paradigm  and  details  on  the 
testing  model  is  provided*    An  attempt  is  made  to  pinpoint  the  decision 
points  in  each  model,  spelling  out  the  consequences  of  the  various 
possible  actions  in  relation  to  each  of  the  ''possible  true  states  of 
nature*" 

The  discussion  of  the  models  is  based  on  descriptions  found  in 
books,  papers,  and  reports;  on-site  visits;  i.nd  meetings  with  many  of 
the  developers*     It  should  be  noted  however  that  programs  are  often 
Implemented  by  teachers  quite  differently  than  they  are  reported  in 
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the  liceracure*    Also,  ic  should  be  reiDembered  that  these  programs  are 
constantly  changing;  hence,  It  Is  possible  that  certain  features  of 
the  models  are  not  exactly  as  they  are  described  here*    In  particular, 
it  is  our  impression  that  PLAN  is  being  implemented  in  a  way  quite 
different  from  how  it  has  been  described  in  the  literature.    This  is 
because  Westinghouse  Learning  Corporation  has  now  taken  over  the 
development  and  implementation  components* 

A  second  purpose  was  to  compare  the  three  programs  and  the  four 
component  parts  of  the  testing  model;  namely,  selection  of  a  program 
of  study,  criterion-referenced  testing  on  the  unit  objectives, 
assignment  of  instructional  modes,  and  final  year-etid  assessment* 

A  final  purpose  was  to  briefly  outline  several  promising  lines  of 
research  in  connection  with  the  testing  methods  and  decision  procedures 
for  individualized  instructional  programs. 
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11,    Individually  Prescribed  Instruction  (n*i) 
2 , 1  Background 

The  Learning  Research  and  Develo.Tment  Center  (LRDC)  at  the  University 
of  Pittsburgh  initiated  the  Individually  Prescribed  Instruction  Project 
during  the  early  1960*s  at  the  Oakleaf  School  in  cooperation  with  the 
Baldwins-Whitehall  Public  School  District  near  Pittsburgh*    Major  contributors 
to  the  project  over  the  years  include  Robert  Glaser>  John  Bolvin,  C. 
Lindvall,  and  Richard  Cox.    Initial  activities  concentrated  on  producing 
instructional  materials  and  training  materials*    More  recently,  research 
and  evaluation  activities  have  assumed  an  increasingly  important  role  In 
Center  activities* 

As  of  1972  the  IPI  program  was  being  implemented  in  over  250  schools 
around  the  country*    Distribution  of  materials  and  other  information  on  the 
program  is  managed  by  Research  for  Better  Schools,  Inc*,  a  United  States 
Office  of  Education  Regional  Laboratory  located  in  Philadelphia*  At 
present^  instructional  materials  are  available  in  elementary  mathematics, 
reading,  science^  handwritings  and  spelling* 

2*2    Description  of  the  Instructional  Paradigm 

While  we  will  discuss  the  instructional  paradigm  and  the  corresponding 
test  model  in  the  context  of  the  IPI  mathematics  program^  the  procedures^ 
techniques^  etc*^  described^  are  in  no  way  limited  to  that  content  aroa* 
In  facts       should  be  noted  that  the  mathematics  program  as  implemented 
is  probably  somewhat  different  from  what  we  describe  here^  since  the  LRDC 
is  constantly  refining  and  improving  the  program  (Lindvall^  personal 
communication)*    Fortunately^  for  our  purposes  the  basic  structure  of  the 
program  remains  as  described* 
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It  Is  instructive  first  of  all  to  describe  the  structure  of  the 
m^tliemaLics  curriculum.    Cooley  and  GJaser  (l969)  report  that  the  mathe- 
matics curriculum  consists  of  430  specified  instructional  objectives* 
These  objectives  arc  grouped  into  88  units.     (In  the  1972  version  of  the 
program  there  were  359  objectives  organized  into  7l  units.)    Each  unit  is 
an  instructional  entity  which  the  student  works  througli  at  any  one  time, 
there  are  5  objectives  per  unit,  on  the  avetagej  the  range  being  I  tc^  14. 
A  collection  of  units  covering  different  subject  areas;  in  mathematics  com- 
prises a  ]evel;  the  levels  may  be  tliouglit  of  a^  roughly  comparable  Lu  school 
grades*    i'or  illustrative  purposes.  Table  2.2,1  presents  the  iiumher  ol 
objectives  for  each  unit  in  the  iPx  mathematics  eurrieulnmt 

The  teacher  is  faced  with  the  problem  of  locating  for  each  student, 
that  Voint  in  the  curriculum  where  he  can  most  profitably  begin  instriic- 
tioni    Also,  the  teacher  is  responsible  for  the  continuous  diagnosis  of 
pupil  demonstrating  proficiency  ill  each  skill  prescribed  in  his  particular 
Instructional  sequence  as  he  moves  along. 

At  the  beginning  of  each  school  year  the  teacher  places  the  student 
within  the  curriculum;  that  is,  he  identifies  the  units  in  each  content 
area  for  which  instruction  is  required.    After  completing  the  gross  plnco- 
mentj  a  single  unit  is  selected  as  the  starting  point  for  instruction,  and 
a  diagnostic  instrument  administered  to  assess  the  student *s  competencies 
on  objectives  within  the  unit.    The  outcome  of  the  unit  test  is  information 
appropriate  for  prescribing  in55truction  on  each  objective  in  the  unit. 
In  addition  it  is  also  necessary  to  select  the  particular  set  of  resources 
for  the  student.     In  theoryj  resources  that  match  the  iudividtial  *s  **ic;ii"n- 
ing  style'*  are  selected.    Within  each  unit,  there  ^ire  short  Lcstsi  tc* 
monitor  the  student's  progress,    finally,  upon  completion  of  inttial  In- 
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Table  2.2.1 

Number  of  Objectives  for  Each  Unit  in  the 
IPX  Mathematics  Curriculum 


Content  Area 

Levels 

A 

B 

c 

D 

E 

F 

G 

H 

Numeration 

12 

10 

8 

8 

8 

3 

8 

A 

Place  Value 

3 

5 

10 

7 

5 

2 

1 

Addition 

3 

10 

5 

8 

6 

2 

3 

2 

Subtraction 

6 

3 

1 

3 

1 

Multiplication 

8 

11 

10 

6 

3 

Division 

7 

7 

9 

5 

5 

Combination  of  Proc«sse5 

6 

5 

7 

5 

6 

Fractions 

3 

2 

6 

6 

14 

5 

2 

Money 

6 

1 

Time 

3 

2 

7 

9 

5 

3 

1 

Systems  of  Measurement 

3 

5 

7 

3 

2 

Geometry 

2 

2 

3 

9 

10 

7 

9 

Special  Topics 

1 

3 

3 

5 

4 

5 

luced,  by  permission,  from  Lindvall,  Cok,  and  Bolvin  (1970)^ 
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structlon  in  each  unlt>  assessment  and  diagnostic  testing  takes  place* 
In  the  next  section,  we  review  the  tests  and  the  mechanisms  for  making  these 
decisions*    Suffice  to  say  here  that  it  has  been  found  that  teachers  differ 
in  the  extent  to  which  they  follow  prescribed  decision-making  rules  (Lindvall, 
Cox,  and  Bolvin,  1970)* 

2*3    Details  of  the  Testing  Model 

Various  reports  over  the  last  couple  of  years  have  dealt  with  the 
testing. model  and  its  development  (Lindvall,  Cox,  and  Bolvin,  1970;  Glaser 
and  Hitko,  1971;  Cox  and  Boston,  1967)*    A  flow  chart  of  the  testing  model 
is  presented  in  Figure  2*3*1*    To  monitor  a  student  through  the  program 
the  following  tests  ar^  used:    placement  tests,  unit  pretests,  unit  post- 
tests,  and  curriculum- embedded  tests*    All  of  the  tests  are  criterion- 
referenced  with  performance  on  the  tests  compared  to  performance  standards 
for  decision-making  4 

How  sophisticated  is  the  decision-making  process  utilizing  the  scores 

from  the  various  tests?    According  to  Glaser  (1968); 

At  the  present  stage  of  our  knowledge,  the  decision  rules 
for  going  from  measures  of  student  performance  to  instruc- 
tional prescriptions  may  not  be  very  complex,  but  little 
is  known  about  the  amount  ot  complexity  required,  although, 
the  individual  monitoring  of  student  performance  provides 
us  with  a  good  data  base  to  study  this  process* 

Promising  developments  in  the  last  couple  of  years  Include  increased 

knowledge  about  constructing  and  evaluating  criterion-roforenced  lests^  Also, 

the  research  on  branched  testing  strategies  (Ferguson,  1969,  1971)  has  much 

potential  for  improving  the  efficiency  o£  thu  testing  model*    This  second 

point  will  be  discussed  in  greater  detail  in  a  iaUer  section. 

Placement  Tests 

Wtien  a  new  student  enters  the  program,  it  is  necessary  lo  pi*KH^  tlu' 
student  at  the  appropriate  level  of  instruction  iu  each  o£  tho  content  afcas. 
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Figure  2.3.1.  Flow  chart  of  steps  in  monitoring  student  progress  in  the 
IPI  program.    (Reproduced,  by  permission,  from  Lindvall  and  CoXf  1969.) 
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[Glaaer  and  Nltko  (197l)  called  this  stage-one  placement  testing.] 
Typically,  this  is  done  by  administering  a  placement  Lest  Tohich  covtira 

all  of  the  subject  areas  at  a  particular  level  (see  Table  2*2.1)*    Factors  effect- 
ing the  selection  of  a  level  for  placement  testing  of  a  student  include 
student  age,  past  peiformance,  and  teacher  judgment*    Generally,  the  placement 
test  covers  the  most  difficult  or  mobt  characteristic  objectives  within  each 
arer,.    Placement  tests  are  administered  until  a  unit  profile  Identifying  a 
stident^s  competencies  within  each  area  is  complete.    At  present,  the  somewhat 
arbitrary  80"85%  proficiency  level  is  used  [or  most  teats  in  the  IPI  fiyateni. 

Scores  for  a  student  on  items  measuring  objectives  in  each  unit  and  area 
in  the  placement  test  are  used  to  define  an  individual  progr^im  for  hUn.  The 
standard  procedure  is  to  assign  instruction  on  units  in  which  placement  test 
performance  on  items  measuring  a  few  representativt!  objectives  in  the  units 
is  between  20%  and  ^0%.    If  the  score  is  less  than  20%  for  a  given  unit, 
the  unit  test  in  the  area  at  the  next  lowest  level  is  administered  and  the 
same  criterion  is  applied*    If  he  passes  the  unit  test,  he  receives  instruc- 
tion in  the  unit  in  ttie  next  level.    In  the  case  where  a  student  ha»  a  score 
of  80%  or  over,  he  is  tested  on  the  unit  in  the  area  at  the  next  highest 
level.    [Further  information  is  provided  by  Lindvall,  Cox,  and  Bolvin  (1*)7\)), 
Weisgerber  (197l)  and  Cox  and  Boston  (1967).) 

For  example,  suppose  a  student  were  to  achieve  scores  on  level  L  ol 
60%,  90%,  60%,  60%,  30%,  30%,  25%,  90%,  50%,  10%,  0%,  30%,  30%  in  the  thirteen 
areas  indicated  in  Table  2*2.1.    It  is  likely  that  he  would  be  prescribed 
instruction  at  level  E  in  the  areas  of  nunieration,  addition,  subtraction, 
multiplication,  division,  combination  of  proceasca,  money,  ^jecmtrtry,  and 
special  topics*    He  would  receive  the  level  F  placanenL  tests  in  pLnco 
value  and  fractions.    If,  for  example,  he  scores  60%  and  10%  rospct-t  tvol>\ 
he  would  ret:eive  Instruction  at  ievoi  F  in  pLaco  vaJue  and  probably  at 
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levol  E  in  fractions*  .  He  would  also  be  administered  the  level  D  placement 
tests  In  the  areas  of  time  an^  systems  of  measurement.    If»  for  example^  his 
SCI  res  were  0%  and  40%^  he  would  receive  a  st:ill  lower  placement  test  In 
the  area  of  time  and  would  be  prescribed  Instruction  at  level  D  In  systems  of 
measurement*    If  he  scores  85%  on  the  level  C  placement  test  in  the  area 
of  tirne^  he  would  be  assigned  to  level  D  for  instruction. 

In  order  to  acquire  some  information  on  the  average  length  of  the  tests^ 
the  level  E  placement  tests  of  the  1972  edition  of  the  IPX  program  were  selected 
and  examined*    Analysis  revealed  that  on  the  average  there  are  12  items 
measuring  the  objectives  in  each  area  (with  a  range  of  from  six  to  20). 

In  summary^  we  note  that  the  placement  test  has  the  following  character*^ 
istics:    provides  a  gross  level  of  achievement  for  any  student  in  the 
curriculum,  and  provides  information  for  proper  placement  of  students  lii 
the  curriculum. 

Unit  Pretests  and  Posttests 

Having  received  an  initial  prescription  of  units>  a  student  procetids 
by  taking  a  pretest  for  a  unit  at  the  lowest  level  of  mastery  on  his  profile. 
[Glaser  and  Nitko  (1971)  call  this  stage-^two  placement  testing.]    A  unit 
pretest  includes  one  or  more  items  to  measure  each  objective  in  the  unit.  A 
review  of  the  unit  pretests  and  posttests  in  level  E  revealed  that  the 
approximate  number  of  items  on  a  test  is  37  (the  range  is  from  21  to  6A)  and  thf* 
average  number  of  items  measuring  each  objective  is  six  (the  range  is  from  four 
to  seven)*    Lindvall  and  Cox  (1969)  report  that  the  length  of  a  pretest  Is 
determined  by  the  number  of  objectives  in  the  instructional  unit  and  by  the 
number  of  items  used  to  test  each  objective.    No  fixed  aumbor  ot  Ucin.s  Lo 
measure  each  objective  is  used  becaui^c  of  the  dlvor«t*  naturo  of  iho 
objectives*    I'or  example^  they  note  that>  "an  objective  like — the  pupil 
can  solve  simple  addition  problems  involving  all  number  combluaLiont. — vjill 
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require  more  items  than  would  an  objective  like — the  pupil  must  s<ilect 
which  of  three  triangles  is  equilateral—." 

A  student  is  prescribed  instruction  in  each  objective  in  the  unit  for 
which  he  falls  to  achieve  an  85%  mastery  level.'*'  In  the  case  where  the 
student  demonstrates  mastery  of  each  objective^  he  is  moved  on  to  the  next 
unit  in  his  profile,  where  he  again  takes  a  pretest. 

The  unit  posttests  are  simply  alternate  forms  of  the  unit  pretests  and 
are  administered  to  students  as  they  complete  instruction  on  the  unit.  A 
student  receives  a  mastery  score  for  each  objective  in  the  unit*    He  is 
required  to  repeat  instruction  on  any  objective  where  he  fails  to  achieve 
an  85%  mastery  score.    He  is  directed  to  the  next  unit  in  his  profile  if 
he  demonstrates  mastery  on  each  objective  covered  in  the  unit  posttest. 
Those  who  repeat  instruction  on  one  or  more  of  the  objectives  must  take  the 
unit  posttest  again  before  moving  on  in  their  program. 

In  Summary  I  pretests  and  posttests  are  available  for  each  unit  of 
instruction*    The  proper  pretest  is  administered  on  the  basis  of  student's 
curriculum  profile,  and  learning  tasks  for  each  skill  are  assigned  (or  not 
assigned)  on  the  basis  of  a  student's  performance  on  items  measuring  the 
skill. 

Coinpared  with  students  in  many  other  types  of  mathematics  programs, 
it  is  clear  tl^t  the  student  in  the  IPI  program  spends  more  of  his  time 
taking  tests*    However,  to  some  extent  this  can  be  justified  on  the 
grounds  that  testing  is  an  integral  part  of  the  learning  process  in  the 
IPI  model.    Nevertheless,  there  seems  to  be  good  reason  for  researching 
techniques  to  reduce  testing  time. 


A  mastery  score  on  each  objective  for  a  student  is  calculated  as  the 
percentage  of  items  on  the  test  that  measure  the  objective  that  the  student 
answers  correctly. 
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Hsu  and  Carlson  (1972)  point  out  several  problems  associated  wHli  the 
current  version  of  the  unit  pretests  and  posttests.    The  existing  syslctn 
requires  tltAt  every  objective  be  tested;  hence,  the  time  a  student  spends 
taking  tests  is  considerable.    Also,  because  of  management  and  scoring 
problems,  feedback  to  the  student  on  his  results  is  not  immediate,  further, 
students  are  occasionally  required  to  take  the  same  posttest  on  a  second 
occasion.    This  raises  a  question  about  practice  effects 

One  very  promising  way  to  reduce  the  testing  time  with  the  correlated 
re5sult  of  producing  better  instructional  decisions  is  suggested  in  the 
branched  testing  work  of  Ferguson  (1969,  1971)*    Ferguson  showed  that  by 
using  a  tailored  testing  strategy,  a  computer  terminal  to  monitor  the 
selection  of  test  items,  and  information  on  the  hierarchical  structure  of  the 
items,  he  was  able  to  ^5ignif icantly  reduce  unit  testing  time  without  ;iny 
loss  in  dec is ion -making  accuracy.    A  comprehensive  review  of  the  work  in 
branched  testing  is  out  of  place  here;  suffice  to  say  here  that  major 
contributions  to  the  area  include  Ferguson  (1969,  1971),  and  Lord  (1970)- 
A  review  of  some  of  the  work  in  the  area  is  provided  by  Bock  and  Wood 
(1972). 

Curriculum-Embedded  Tests 

As  the  student  proceeds  through  a  unit  of  instruction,  his  progress 
must  be  monitored*    This  is  done  by  curriculum^embedd^ad  tests  (CKT)* 
As  used  in  the  mathematics  IPI  program,  a  CET  is  primarily  a  measure  of 
performance  on  one  specific  objective.    There  are  usually  several  test  items 
to  measure  the  objective^    A  review  of  the  CETs  in  level  E  of  the  program 
revealed  tliat  there  are  on  the  average  about  three  items  measuring  the  prlmiiry 
objective  covered  in  the  CET.    The  range  is  from  two  to  five*    Tf  a  sjtudeiit 
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receives  a  score  o£  83%,  he  is  permitted  to  move  on  to  the  next  prescribed 
objective.    Otherwise,  he  is  sent  back  for  additional  work  and  then  he 
takes  an  alternate  form  of  the  CET  when  he  is  ready* 

A  Secondary  purpose  of  the  CET  is  to  pretest,  in  a  rough  way,  the  next 
objective  in  the  learning  sequence.     (Objectives  in  a  unit  are  arranged 
into  a  learning  sequence.)    Students  may  pretest  out  of  the  next  skill  in 
the  Sequence  by  achieving  83%  or  higher  on  the  short  test  which  makes  u^j 
the  Second  part  of  the  CET  and  on  part  one  of  the  CET  for  that  skill.  It 
would  appear  from  a  review  of  level  E  tests  that  there  are  about  two  items 
measuring  the  secondary  objective.     In  cases  where  a  student  does  not  need 
instruction  on  the  next  skill,  he  can  skip  part  two  of  the  CET  and  move  on 
to  the  part  two  of  the  CET  that  tests  the  next  skill  he  needs  for  his 
program.    This  additional  pretesting  of  an  objective  in  the  CET  givas 
students  a  chance  to  demonsLrate  mastery  of  new  skills  not  specifically 
covered  in  the  instruction  to  that  point  and  to  eliminate  that  instruction 
from  his  program. 

Student  Diagnosis 

Once  the  student  has  been  assigned  to  a  unit  of  instruction  and  the 
objectives  for  which  he  needs  instruction  have  been  identified  by  the  unit 
pretest,  there  still  remains  the  problem  of  deciding  which  of  several 
instructional  methods  is  "optimal*'  for  him.    That  is,  of  the  available 
Instructional  methods  for  a  particular  instructional  unit,  in  which  of  them 
would  a  student  with  a  known  background  in  the  program  and  specific  goals, 
interests,  and  aptitudes  stand  the  %est**  chance  of  learning  the  material? 
Glaser  and  Hitko  (1971)  call  this  a  diagnostic  decision. 
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III*    Program  for  Learning  in  Accoivlance  with  Meeds  (PLAN) 
3,1  Background 

Project  PLAN  is  a  major  ungraded^  computer-supported  individualized 
instruction  program  in  education  developed  by  the  American  Institutes  for 
Research  over  the  last  seven  years*    (For  background,  se*^.  Weisgerber,  1971*) 
The  project  was  initiated  by  John  Flanagan  to  handle  many  of  the  short- 
comings of  our  educational  system  as  revealed  by  Project  TALENT  (Flanagan, 
et  al* ,  1964)* 

The  PLAN  program  is  currently  being  used  in  over  70  schools  with  more 
than  35,000  students  in  f^rades  one  through  twelve*  Instructional  materials 
are  available  in  four  areas:  social  studies,  languaj^e  arts,  mathematics, 
and  science*  Westinghouse  Learning  Corporation  is  now  responsible  for  the 
monitoring  and  marketing  of  Project  PLAN  materials.  They  also  operate  the 
computer  installation  necessary  for  the  proper  functioning  of  Project  PT>AH 
in  a  school* 

Unfortunately,  the  implementation  of  the  model  in  1972-73  involves  far 
fewer  features  than  was  originally  described  by  the  proponents  of  the  program 
a  few  years  ago*    Nevertheless,  we  will  describe  the  more  elaborate  version 
of  the  program  in  this  paper « 

3^2    Instructional  Paradigm 

The  basic  unit  of  instruction  in  PLAN,  called  a  module,  is  an  instruc- 
tional package  :jade  up  of  about  five  behavioral  objectives.    Tt  normally 
takes  a  student  about  two  weeks  to  complete  a  module  of  Instruction.  Also, 
there  are  many  objectives  classified  at  the  higher  levels  of  Bloom's  (1,956) 
taxonomy  that  do  not  fit  nicely  into  the  regular  modules.    These, nre 
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named  module-set  objectives^  and  examples  Include  concept  development  and 
problem-solving  skills*    They  are  worked  into  the  regular  modules  and  prog- 
ress is  measured  by  PLM^  ac;iievement  tests  administered  periodically  through- 
out the  program*    According  to  Rhetts  (1970)  there  are  more  than  1100 
modules  in  PLAN.    For  each  module^  there  are  several  different  teacher- 
learning  units  (TLU)  assigned  individually  on  the  basis  of  aptitudes » 
interests^  learning  style>  etc*    All  modules  in  the  secondbry  school 
curricula  are  coded  as  to  whether^  1)  they  are  part  of  a  state  or  local 
requirement »  2)  essential  for  a  given  educational  or  occupational  area» 
3)  highly  desirable  for  that  area»  4)  essential  for  minimum  functioning 
as  a  citizen^  5)  highly  desirable  for  all  citizens  to  know»  or  6)  would 
make  the  student  a  particularly  well  informed  citizen* 

TLU*s  are  coded  according  to:    1)  reading  difficulty^  2)  degree  to  which 
it  requires  teacher  supervision^  3)  its  media  richnefss,  4)  degree  to  which 
it  requires  social  involvement  and/or  group  learning  activities^  5)  the 
amount  of  reading  involved^  and  6)  variety  of  activities  in  the  module. 
There  are,  on  the  average,  two  TLU's  for  each  module.    Along  the  lines  of 
Dunn  (1970),  we  will  describe  the  most  complex  version  of  the 
program"the  version  currently  being  used  in  the  secondary  school. 

At  the  beginning  of  each  ye^r,  a  program  of  study  is  prepared  for  each 
student*    This  includes  a  list  of  modules,  suggested  TLU^s,  and  a  recommended 
sequence  in  the  four  content  areas.    To  really  provide  intli visualized 
instrixtion,  it  is  necessary  to  know  about  student  needs,  goals,  abilities, 
and  interests  and  to  use  the  information  in  developin}^  a  prop,ram  of  study 
(POS)  for  him.    As  part  of  the  PLAN  system  then,  the  following  ii^formatJon 
is  collected : 
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1,  parent  and  student  educational  goals 

2,  parent  and  student  vocational  aspirations 

3*    student  level  of  achievement  and  vocational  interests 
4«    student  abilities  (such  as  reading  comprehension  and  arithmetic 
reasoning) 

5.  past  performance  of  student  in  program 

6.  student's  learning  style* 

A  variety  of  questionnaires  and  testing  instruments  have  been  developed 
to  collect  the  above  information* 

Abilities  are  measured  each  year  with  the  Developed  Abilities 
Performance  Test  (DAFT) ^    This  test  consists  of  18  scales  (see>  for  examPle> 
Jung»  1970)  such  as  those  to  measure  arithmetic  reasonings  reading 
comprehension >  abstract  reasoning^  mechanical  comprehension^  and  ingenuity* 

On  the  basis  of  the  above  information^  a  prof^ram  is  developed  and  the 
student  is  monitored  through  it  by  continuous  module  posttestlng  and  PLAN 
achievement  testing.    Let  us  look  now  at  the  testing  ptiase  of  the  pro>\ram 
in  more  detail » 

3*3    Testing  Model  Details 

Within  a  PLAN  school^  there  exists  a  multitude  of  decisions  to  make  on 
each  student.    These  include  development  of  a  program  of  study,  peric^dic 
assessment  of  module-set  objectives^  performance  on  the  modules;  of 
instruction^  assignment  of  TLU^s^  and  yearly  monitoring  of  important 
skills*    The  major  decision  points  are  shown  in  Figure  3*3*1* 
Unfortunately^  there  is  little  available  information  on  how  these 
decisions  are  made* 
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Developed  Aptitude  Performance 
Tests  taken 


One  module  selected 
for  study 


Module  Pretest 
(optional) 


Pass  all  skills 


Fail  one  or  more 
skills 


Prescription  developed: 
relevant  TLU 


^fodule  Posttest 
taken 

Pass  all 
skills 


Fall  one  or 
more  skills 


Figure  3.3.1    Flow  chart  of  steps  in  monitoring  student  progress  in 
Project  PLAN. 
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Development  of  a  Program  of  Stud^ 

On  the  basis  of  DAPT  scores  which  are  matched  to  Project  TALENT 
data  of  people  in  different  occupations,  the  students  and  parents  sel'ict 
a  long  range  goal  [(LRG)  (one  of  12  families  of  occupations)]*  Information 
on  the  long  range  goal  along  with  parent  and  student  information  described 
in  the  last  section  is  used  to  develop  a  program  of  study.    The  DAPT  is 
also  used  in  the  determination  of  the  number  of  modules  a  student  will 
study  in  a  year*    Jung  (1970)  reports  that  on  the  basis  of  weights  derived 
from  regression  analyses,  a  quota  is  identified  for  each  PLAN  student 
in  each  subject  area,    ^fodules  are  then  assigned  to  him  on  the  basis  of 
his  LRG  group  mecbership  until  this  quota  is  filled. 
Developed  Aptitude  Performance  Tests 

These  test:^  are  given  at  the  beginning  of  each  school  year.  Informa- 
tion on  the  length,  kinds  of  test  items,  reliability  and  valicJity  does  not 
appear  to  have  been  published*    Also,  we  do  not  know  whether  a  different 
version  cf  the  test  is  used  in  each  year,  or  whether  the  same  version  is 
used  for  several  years.    Regardless,  unless  comparability  of  the  score 
scales  for  the  different  versions  has  been  carefully  done,  we  doubt  whether 
the  change  scores  (for  individuals  or  groups)  on  each  variable  from  year 
to  year  have  very  much  meaning. 
PLAN  Achievement  tests 

Mastery  of  the  Taodule**set  objectives  is  measured  at  specific  points 
in  the  curriculum  using  PLAK  achievement  tests*    However,  we  are  also 
unclear  on  the  make-up  of  the  PLAK  achievement  tests.    Apparently,  tliey 
are  measured  at  "specified  points"  in  the  curriculum  and  the  formciL  or 
these  tests  is  sometimes  something  other  thau  the  paper  anc!  pencil  V(iriet>. 
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Module  Tests 

When  the  student  feels  he  has  mastered  the  materials  covered  in  a 
module^  he  can  take  a  criterion-referenced  module  posttest  which  has  on 
it  several  items  measuring  each  objective  In  the  module.    The  items  are 
presented  usually  in  a  selectioii  format  to  facilitate  computer  scoring. 
On  the  basis  of  his  performance^  the  r.omputer  using  built-in  decision  rules 
makes  one  of  four  decisions.    If  he  answers  all  items  correctly^  he  is 
giver,  a  ^'complete"  on  the  module  and  the  computer  print  out  tells  him  where 
to  go  next.    If  he  makes  a  "few"  errors,  he  Is  given  a  result  of  ''Student 
Review",    The  computer  specifies  his  performance  on  each  objective  and 
indicates  the  ones  he  should  review  before  beginning  his  next  module. 

Students  wlio  miss  a  large  number  of  items  on  the  test  but  stilj  score 
high  enough  to  pass,  receive  a  result  of  "Teacher  Certify",    lie  is  instructed 
by  the  teacher  on  which  objectives  to  review  and/or  restudy.    He  is  noL 
given  his  next  module  until,  in  the  judgment  of  the  teacher,  he  has  mastered 
all  of  the  objectives.    An  alternative  is  to  have  the  student  repeat  the 
module  posttest.    The  fourth  possibility  is  student  failure  to  pass 
the  test.    In  this  situation,  he  is  instructed  to  restudy  the  nodule  with 
the  same  TLU  or  another.    In  the  case  where  he  nif;ses  the  test  again,  the 
teacher  intervenes  and  takes  some  appropriate  action  to  cJear  up  the  problem. 
Assignment  to  Instructional  Mo'des  . 

The  basic  problem  was  described  in  a  discussion  of  the  TPI  program, 
i,e,,  what  particular  instructional  mode  (or  in  this  case,  TLU),  should  the 
student  take  to  study  the  module  so  as  to  maximize  his  changes  of  learning 
the  material?    Dunn  (1970)  notes,  "that  the  computer,  from  a  complex 
set  of  decision  rules,  matches  the  student  with  specific  TL'Vs",    We  wonder 
what  those  rules  would  be,  particularly  since  there  is  no  theory  of  instruction 
to  guide  in  developing  optimal  assignment  rules.    To  this  point  in  time 


25 


educational  psychologists  have  only  been  able  to  find  a  handful  of 
interactions  between  background  variables  and  instructional  method*  A 
partial  answer  is  provided  by  Weisgerber  and  Rahmlow  (1971).    Tliey  noted 
that  teacher-learning  units  are  based  upon  different  assumed  learning 
styles  of  students  ^nd  are  guided  by  a  philosophy  of  education  (Flanagan^ 
1970)  and  a  theory  of  learning  (Gagng^  1965)* 
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IV*    Mascery  Learning 

Background 

The  mastery  learning  concept  was  introduced  to  American  Schools  In  tlie 
the  l920*s  with  the  work  of  Washburne  (1922)*    However^  because  technology 
was  not  developed  to  the  point  that  the  program  could  operate  efficiently, 
interest  in  the  concept  steadily  diminished  until  it  was  revived  in  the 
form  of  programmed  instruction  in  the  late  1950* s.     (Programmed  instruc- 
tion was  an  attempt  to  provide  students  with  instructional  materials 
that  would  allow  them  to  move  at  their  own  pace  and  receive  constant 
feedback  on  their  level  of  mastery*)    The  work  by  Carroll  (3963,  1970) 
and  Bloom  (1968)  and  Bloom^s  students  (Block,  1971;  Airasian,  1971  and 
others)  was  instrumental  in  bringing  mastery  learning  to  the  attention  of 
instructional  designers  and  researchers* 

Since  Bloom*  s  paper  In  1968,  a  great  deal  of  research  has  been  conducted; 
and  the  results  suggest  that  the  mastery  learning  model  "can  be  easiJy  and 
inexpensively  implemented  at  all  levels  of  education  and  in  subjects 
ranging  from  arithmetic  to  philosophy  to  physics  (Block,  1970)*  Thfe 
model  has  been  used  now  with  more  than  20*000  students* 

A. 2    Instructional  Paradigm 

This  model  is  quite  different  from  IPX  and  PLAN  in  chat  it  attempts 
to  individualize  instruction  within  a  group-based  instructional 
environment*    The  curriculum  is  organized  into  units  of  Instruction 
defined  by  homogeneous  clusters  of  behavioral  objectives.    For  each 
unit  one  or  more  criterion-referenced  tests  is  used  to  measure  mastery* 
Individualization  Is  handled  via  supplemental  materials,  feedback, 
and  corrective  techniques  applied  to  students  who  do  poorly  on  the 
post tests* 
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Mayo  (1970)  in  describing  the  mastery  learning  model  notes  that: 

1.    Students  are  made  aware  of  course  and  unit  expectations,  so  that 

they  view  learning  as  a  cooperative  rather  than  as  a  competitive 

venture « 

2*    Standards  of  mastery  are  set  in  advance  for  the  students,  and 
grading  is  in  terms  of  absolute  performance  rather  than  relative 
performance* 

3*    Short  diagnostic  tests  are  used  at  the  end  of  each  instructional 
unit « 

4.    Additional  learning  is  prescribed  for  those  who  do  not  demonstrate 
unit  mastery* 

5*    Additional  time  for  learning  is  prescrihed  to  students  who  seem 
to  need  it* 

The  mastery  learning  mode)  is  less  impressive  in  scope  than  PLAN,  and 
the  requirements  for  an  effective  testing  plan  are  less  stringent  than  with 
IPX  or  PLAW.    Features  of  mastery  learning  appear  to  be  that  it  is  easily 
itnplementable,  does  not  require  the  use  of  a  computer,  and  is  appropriate 
for  almost  any  content  area*    Also,  if  mastery  learning  is  carried  out 
properly,  previous  research  suggests  that  students  will  achieve  higher 
scores  and  have  more  interest  in  school  and  a  better  attitude  toward  school* 
Unlike  the  other  two  models,  with  mastery  learning  much  of  the  work  has  been 
on  research  related  to  the  correctness  of  the  model  of  school  learning. 
An  extensive  number  of  content  areas  have  been  studied. 

It  should  be  noted  that  there  are  many  variations  on  the  basic  m^istery 
model  as  originally  proposed  by  Bloom  (1968)*    Some  of  them  ate  summiuist'd 
by  Block  (1971),  and  an  example  would  be  tha  work  of  Kim  (1971)* 
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4*3    Test  Model  Details 

Block  (1971)  notes  that,  **To  individualize  instruction  within  the 
context  of  ordinary  group-based  instruction,  mastery  learning  relies 
heavily  on  the  constant  flow  of  feedback  information  to  teacher  and 
learner***    It  does  not  seem  however  that  there  is  as  much  testing  in 
mastery  learning  as  in  I?I  or  PLAN*    A  flow  chart  of  the  testing  component 
is  shown  in  Figure  4*3.1* 

The  mastery  learning  testing  model  as  described  by  Alrasian  (1971) 
represents  a  special  case  of  the  IPX  testing  program*    There  is  no  place- 
ment testing,  and  unit  pretesting  and  curriculum-embedded  testing  are  not 
emphasized.    Unit  posttestlng  and  final  assessment  represent  the  two  P»aior 
kinds  of  testing  in  the  program.    In  the  spirit  of  Scriveii  (1967),  these 
two  areas  are  known  as  formative  and  summatlve  tests*    It  shcmld  be  n<>ted, 
however,  that  formative  tests  or  unit  posttests,  as  they  are  Called  In  IPX, 
are  not  used  for  grading*    They  are  used  for  diagnosing  learninf^  diffictiltics 
only. 

Formative  Tests 

A  formative  test  Is  designed  to  cover  the  objectives  over  a  sliort  unit 
of  instruction  in  the  mastery  learning  program.     It  is  used  to  (ietermliu'. 
whether  or  not  a  student  has  mastered  the  material  and  to  .serve  a±;  a  hasif^ 
for  prescribing  supplemental  work  in  areas  where  the  student  is  we;ik 
(Airasiani  1971)*    Implementers  of  the  mastery  learning  model  have  set 
the  passing  standard  anywhere  from  75%  to  100%*    There  is  no  set  number  of 
items  or  format  suggested  to  measure  each  objective;  however^  there  is  a 
suggestion  that  instructional  decisions  are  made  on  the  basis  of  responses 
to  individual  items* 

The  formative  tests  in  mastery  learning  represent  the  key  to  indi- 
vidualizing instruction  since  it  is  on  the  basis  of  these  scores  thut 
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Unit  in 
selected 


program 
for  study 


> 

Group-based  ] 
on  the  unit 

Instruction 
objectives 

Unit  Posttest  ^ 
taken  (Formative  Test) 

/  Pass 

all  \ 

I     skills  / 

Free  study 
time,  tutoring 
others,  etc* 


Fail  one  or 
more  skills 


Prescription  developed: 
use  of  alternative 
resources 
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Yes 


Final  assessment: 
Summative  Teat 


Figure  4*3*1    Flow  chart  of  steps  in  monitoring  student  progress  in  a  typical 
version  of  a  mastery  learning  model. 
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Individualization  of  Instruction  can  take  place*    Units  are  kept  small 
so  that  unit  testing  takes  place  frequently  to  Increase  the  effectiveness 
of  the  Individualization  of  Instruction  component  of  the  program. 
Summatlve  Tests 

The  primary  purpose  of  the  summatlve  Lest  In  the  mastery  learning 
model  is  to  grade  students  on  the  basis  of  their  achievement  of  course  objec- 
tives*   The  Items  In  the  test  are  keyed  to  objectives  and  representative  of 
the  pool  of  course  objectives*    A  criterion-referenced  Interpretation  of 
the  scores  Is  recommended*    It  Is  proposed  that  cutting  points  be  located 
on  the  ability  continuum  ^nd  grades  should  be  assigned  on  the  basis  of  a 
student^s  position  on  the  continuum  and  not  relative  Co  other  students  in 
the  course*    A  norm-referenced  interpretation  of  the  scores  is  also  ptissiM^* 
Final  Costif^ents 

Mastery  learning  Is  probably  the  least  different  from  trnttitlonaJ 
instructlor  since  the  principal  instruction  is  always  group-based  and 
final  grades  are  assigned*    (lloweveri  it  is  expected  that  because  of  various 
features  built  into  the  program  that  the  final  assessment  testing  will  not 
be  as  threatening  a  situation  for  the  student  as  it  is  in  more  traditional 
programs*)    Differences  with  traditional  instructional  models  Include 
features  such  as  individual  pacing,  and  the  big  difference  is  the  use  of 
frequency  tests  on  small  units  of  instruction  to  diagnose  learning  problems. 
Important  features  are  the  feedback/correctlng-revlew  techniques*  It 
would  appear,  however,  that  there  is  little  in  the  way  of  sophistication 
concerning  the  testing  model.    For  example,  there  appears  to  be  no 
guidelines  for  determining  the  optimum  number  of  items  to  measure  each 
objective  on  a  unit  posttest*    An  exception  is  the  excellent  work  of 
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Block  (1970)  in  investlgatlngi  among  ocher  things,  the  problem  of  setting 
cutting  scores  on  criterion-referenced  tests  to  separate  students  into 
two  groups — masters  and  non-masters*    His  results  suggest  that  setting 
cutting  scores  high  (95%)  may  be  best  for  cognitive  learning  but  in  the 
long  run  positive  attitudes  and  interest  in  the  subject  are  less  likely 
to  develop*    With  a  reduction  in  the  cutting  score  to  85%  there  was  a 
reduction  in  cognitive  learning,  but  selected  affective  outcomes  were 
maximized » 
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A  Comparison  o£  Che  Testing  Models 

5.1  Introduction 

In  Che  three  previous  sections  we  have  highlighted  the  basic  testing 
3nd  decisionmaking  features  In  three  Individualized  Instructional 
programs—IPI,  PLAN,  and  Mastery  Learning.    Within  all  three  models, 
Instruction  Is  self-paced  although  mastery  learning  Is  somewhat  more 
structured  since  the  Initial  Instruction  on  a  unit  Is  group^iaced.  With 
each  of  the  models,  the  content  Is  organized  Into  units  or  modules. 
C«)nerally,  In  IPI  and  ML  the  student  Is  expected  tc  demonstrate  mastery 
on  all  the  units  before  completing  the  program  of  study  although  by  his 
performance  on  unit  pretests.  It  Is  possible  for  him  to  avoid  Instruction 
on  any  of  the  units.    (One  variation  that  does  cone  up  Is  the  availability 
of  '^enrichment  materials^*  which  are  an  optional  part  of  the  curriculum.) 
In  PLAN,  at  any  grade  level  there  are  far  more  units  than  any  student  could 
or  would  ever  want  to  master.    Thus,  It  Is  first  of  all  necessary  to 
define  a  content  domain  of  study  for  each  student. 

In  the  remainder  of  the  section,  we  shall  limit  discussion  to  testing 
and  decisionmaking  Issues.    In  order  to  develop  a  framework  £or  the 
discussion,  we  have  chosen  to  focud  on  the  following  Issues: 

1)  selection  of  a  program  of  study; 

2)  criterion-referenced  testing  on  the  unit  objectives; 

3)  assignment  of  instructional  modes; 

4)  final  year-end  assessment. 

These  represent  the  extent  o£  the  decision  paradigms  within  the 
three  models.    The  importance  and  sophistication  used  in  handling  each 
component  varies  from  one  model  to  another. 
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5.2    A  Compendium  of  Decision  Paradigms 
Selection  of  a  Program  of  Study 

A  program  of  study  is  that  collection  of  units  which  a  curriculum 
designer  deems  necessary  for  the  appropriate  education  of  the  student. 

All  tUr'ie  models  are  designed  for  utilization  with  a  curriculum 
defined  in  terms  of  behavioral  objectives  arranged  into  blocks^  units>  or 
modules  around  a  common  topic  or  theme.    Generally  in  IPI  and  ML>  students 
are  expected  to  demonstrate  mastery  in  all  of  the  available  course 
objectives.    The  available  course  objectives  define  the  program  of 
study  for  the  student*    However,  on  the  basis  of  high  pretest  results 
students  may  avoid  instruction  of  selected  units  of  instruction. 

In  PLAN,  each  student  receives  a  unique  program  of  study*    The  more 
advanced  the  students  the  more  varied  their  programs  of  study  become. 

For  reasons  described  above,  selecting  a  program  of  study  for  a 
student  in  IPI  or  Mastery  Learning  is  relatively  easy*    The  decisions  to 
be  made  reduce,  basically,  to  determining  whether  students  have  mastered 
particular  objectives*    They  will  receive  instruction  only  on  course 
objectives  they  have  not  mastered*    In  IPI,  placement  tests  are  used 
to  determine  the  level  of  instruction  in  each  area  for  the  students* 
Here  the  error  of  giving  the  student  credit  for  units  he  has  not  mastered 
(a  f alse^positive  error)  seems  t^o  be  somewhat  more  serious  than 
mistakenly  assigning  him  to  instruction  he  does  not  need  (a  false- 
negative  error) 4    This  follows  since  a  student  has  a  second  chance  to 
demonstrate  mastery  of  the  objectives  in  a  unit  through  the  unit  pretes*' 
if  he  is  mistakenly  assigned  to  study  a  unit  he  has  already  mastered* 
On  the  other  hand,  to  incorrectly  assign  credit  for  mastering  a  unit 
to  a  student,  particularly  if  it  is  an  important  unit,  will  plaRue  him 
in  his  future  studies* 
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In  theory  at  least  in  the  PLAN  prop.ran^i  developing  a  profiram  of  study 
is  a  complete  affair.    Done  once  a  year  it  requires  a  wealth  of  information 
described  in  section  2,3  to  develop  the  program^    The  danger  of  locatinp, 
a  student  in  the  wro'ig  program  because  of  mlsjudftment  on  the  part  of  the 
parents*  teachers,  or  the  student  or  because  of  a  "less  than  100%  prediction 
system"  are  great;  however,  this  is  the  same  risk  we  take  with  selection  of  a 
progratsi  in  a  traditional  school*    This  is  particularly  serious  in  the  high 
school  where  there  is  more  choice  than  in  the  elementary  school  programs* 
However,  the  flexibility  of  the  PLAN  program  makes  switching  from  one 
program  to  another  easier* 

Criterion-Referenced  Testing  on  the  Unit  Objectives 

There  are  three  kinds  of  testing  appropriate  here:    unit  pretesting « 
unit  posttestlngi  and  curriculum'~embedded  testing.    All  three  kinds  of 
testing  are  used  in  IPX  and  PLAN  although  unit  pretesting  Is  not  stressed 
in  PLAN*    The  possibility  existed  for  all  three  kinds  of  testing  in 
Mastery  Learning;  howeveri  unit  pretesting  is  not  emphasized  and  a  student 
can  avoid  the  curriculum-embedded  testing  by  passing  the  unit  posttest  and 
thus  avoid  the  reTuedial  instructional  materials ^     (Also*  it  is  quite 
possible  that  curriculum-embedded  tests  are  not  available  in  the  remedial 
materials^ 

Let  us  briefly  look  now  at  the  losses  involved  in  making  different 
kinds  of  decisions*     It  should  be  recalled  that  the  unit  tests  (or  module 
tests)  measure  performance  on  each  objective  or  skill  with  several  items* 
On  the  unit  pretests*  a  student  receiving  credit  for  non-mastered  objectives 
will  likely  be  ^'caught"  on  the  administration  of  the  posttest  and  correct 
instruction  can  be  assigned  at  that  time*    Howeveri  to  the  extent  that 
these  objectives  are  prerequisites  to  others  in  the  unit  we  have  a  case 
of  instructional  mismanagement*     (Perhaps,  this  is  a  place  where  Bayesian 
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statistics  might  be  helpful  In  producing  an  *'improved"  profile  of  scores 
across  objectives  measured  by  the  unit  pretest.    This  would  undoubtedly 
improve  the  overall  decision-malting  accuracy.    Likewise  this  strategy 
could  be  used  on  the  unit  posttestsO 

To  assign  a  student  instruction  on  the  basis  of  pretest  score  results 
to  objectives  which  he  has  previously  mastered  will  undoubtedly  prove  to 
be  frustrating  to  him;  however*  it  should  be  noted  that  the  majority  of 
errors  of  this  type  occur  because  stduents  are  close  to  the  cutting  score. 
Thus,  the  problem  does  not  seem  to  be  one  that  needs  to  be  taken  too 
seriously* 

Receiving  credit  for  non-mastered  objectives  on  the  posttest  to  the 
extent  that  the  objectives  are  prerequisites  to  others  in  future  units  will 
interfere  with  the  rate  of  learning  at  that  point*    This  error  seems  to  be 
less  serious  in  terms  of  program  efficiency  if  the  objectives  are  terminal* 
Falling  to  receive  credit  for  mastered  objectives  would  seem  to  be  less 
serious  since  the  student  could  move  through  the  remedial  materia?-S  quickly 
and  retake  the  test* 

Since  any  decisions  on  the  basis  of  currlculum-^embedded  test  score 
results  affect  the  student  for  only  a  limited  amount  of  time  and  there 
exist  checks  on  any  decisions  with  the  unit  (or  module)  posttest,  there 
is  little  concern  for  developing  more  appropriate  testing  decision  guide*- 
lines  at  this  level* 
Assignment  of  Instructional  Modes 

An  integral  component  of  nearly  every  iP J ivl dualized  instruction  pro-^ 
gram  is  the  feature  whereby  there  exists  several  alternate  instructional 
modes  for  the  various  units  of  instruction  that  can  be  assigned  in  some 
optimal  way  to  students.    In  theory  anyway^  with  IPI  and  PLAN,  past  perform- 
ance and  background  aptitude  variables  ava  used  to  assist  the  students  in 
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selecting  the  %est**  mode  of  instruction*    With  Mastery  Learning,  this 
feature  can  be  operationalized  following  the  group-abased  instruction  and 
the  unit  posttests*    It  is  at  this  point  that  decisions  on  the  proper 
corrective  feedback  techniques  to  use  need  to  be  made» 

Investigators  of  th^  possible  interactions  between  instructional  methods 
and  aptitudes  are  conducting  what  has  been  termed  Aptitude-Treatment 
Interaction  (ATI)  research  (Cronbach,  1967)*    Disappointing  is  the  fact  that 
while  nearly  all  developers  of  individualized  programs  include  this  feature  of 
utilizing  ATI  results  in  assigning  instruction^  there  are  few  real  demon- 
strations of  significant  interactions  between  aptitudes  and  instructional 
modes  (Brncht,  1970;  Cronbach  and  Snow,  1969)*    Authors  such  as  Claser 
(1972)  have  attanpted  to  explain  these  results  and  suggest  some  new 
directions  for  this  line  of  inquiry*    However,  it  woul i  appear  that  we  are 
far  from  a  "theory  of  instruction^^  to  guide  the  instructional  decision  maker 
in  the  assignment  of  "optimal"  instructional  modes  to  students. 

The  benefits  (assuming  equal  treatment  costs)  of  the  ATT  classifica- 
tion scheme  for  improving  the  quality  of  instruction  depend  directly  on 
the  differences  among  the  slopes  of  the  regression  lines  for  predicting; 
criterion  scores  with  different  aptitude  variables  in  the  different  iiit;tr\ic* 
tional  modes^    The  bigger  the  difference  in  slopes  the  {>.r^ater  Js  the 
potential  benefit  to  the  student  for  assigning  one  instructional  mode  or 
anotheri    However,  in  looking  at  the  overall  benefits  and  losses  of  such  a 
syst€OT*  it  would  seen  that  the  appropriate  baseline  for  comparative  ^rurpui:**^ 
would  need  to  be  data  derived  from  a  traditional  instructiona]  prograuu 
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Final  Year-End  Assessment 

This  particiilar  feature  seems  to  be  handled  in  much  the  same  way  in 
IPI  and  PLAN.    Information  is  reported  on  the  number  and  nature  of  units 
that  2  3t\iiJent  has  rnastcred*    little  or  no  inforniation  is  provided  by  the 
school  to  students  and  parents  that  could  be  used  for  norm-referenced 
assessment*    In  the  tnastery  learning  model,  a  score  is  reported  to  measure 
achievement  on  the  year-long  activities*    Both  norm-referenced  and 
criterion'-referenced  interpretations  are  possible. 
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VI*    Some  Directions  for  Further  Research 

6*1    Concluding  Remarks 

A  review  of  IPI>  PLAN,  md  Mastery  Learning  programs  as  well  as  many 
other  objective-based  curriculum  programs  not  reported  In  this  paper  re^ 
veals  that  there  are  many  Important  questions  remaining  to  be  answered 
In  regard  to  Individual  assessment  models.    In  this  concluding  section 
a  few  of  the  more  Important  problem  areas  are  discussed* 

In  order  to  develop  an  instructional  model  that  is  sensitive  to, 
indl vidua]  needs,  abilities,  interests,  and  goals  in  a  way  that  will 
allow  the  student  to  maximize  his  learning,  we  need  a  theory  oC  Instruction* 
A  theory  of  instruction  should  set  down  rules  on  the  most  efficient  way 
of  achieving  knowledge  (Bruner,  1964).    This  theory  would  provide  guidelines 
on  how  to 'prescribe  instruction  to  increase  learning*    One  paper  that 
addresses  the  problem  is  Groen  and  Atkinson  (1966).    Current  reports  on  the 
related  topic  of  aptitude-treatment  Interactions  are  by  Cronbach  and  Glei^er 
(1965),  Cronbach  and  Snow,  (1969),  Bracht  (1970),  and  Glaser  (1972). 

In  making  decisions  on  the  basis  of  criterion-referenced  test  scores, 
one  assumes  a  good  match  between  items  and  the  behavioral  objectlvi^s 
they  are  Intended  to  measure*    To  the  extent  that  test  items  do  not 
accurately  measure  the  objectives,  any  decisions  basad  on  test  performance 
will  be  Inaccurate*    To  date  a  satisfactory  methodology  for  item  validation 
does  not  exist  although  several  useful  papers  provide  partial  solutions 
(Dahl,  1971;  Rovinelli  and  Hambleton,  1973)* 

A  theory  of  criterion'-referenced  tests  and  measurements  is  also 
needed  to  guide  the  users  of  the  tests  in  the  context  of  programs 
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described  here.    This  theory  should  probably  be  based  on  a  threshold  loss 
function  rather  than  a  squared-error  loss  function  as  has  been  done  in 
classical  test  theory  (Lord  and  Novlck,  1968;  Hambleton  and  Novlck,  1973)* 
This  theory  would  Include  reliability >  validity,  test  scoring,  and  item 
validation  procedures  for  criterion-referenced  tests*    It  would  also  provide 
guidelines  and  techniques  for  setting  test  lengtlt  and  cutting  scores »  and 
allocating  testing  time*    A  recenc  paper  by  Millaian  (1973)  v^ovides  scnie 

excellent  guidelines  on  this  latter  set  of  problems* 

Another  problem  which  has  to  be  reckoned  with  for  criterion-referenced 
tests  is  an  instance  of  the  bandwidth-fidelity  issue  (Cronbach  and  Gleser> 
1965)*    When  the  total  testing  time  is  fixed  and  there  is  interest  in 
measuring  many  compjitencies,  one  may  be  faced  uith  the  problem  of  whethet 
to  obtain  very  precise  information  about  a  small  numbei  of  skills  or  lesfi 
precise  information  abtmt  many  more  skills*    Time  allocation  algorithms 
(analytical  procedures  Cor  deciding  how  many  itenit;  on  a  test  s^houid  measure 
each  objective)  of  a  rather  different  kind  than  those  presented  by  Woodbury 
and  Novick  (1968),  and  Jackson  and  Novick  (1970)  *jill  be  required*  The 
problem  of  how  to  determine  the  number  of  items  to  measure  each  skill  so 
as  to  maximize  the  percentage  of  correct  decisions  or  *;orje  siwihir  mt-asurc 
of  overall  decision-making  accuracy  on  the  basis  of  test  results  has  yet  to 
be  resolved. 

Estimation  of  mastery  is  a  problem  that  is  encountered  frequently  in 
individualized  ioatructional  programs*    Bayesian  methods  have  been 
suggested  (Hambleton  and  Novick,  1973) >  but  there  has  been  no  empirical 
demonstrations  of  their  usefulness  in  chis  context  nor  are  guidelines 
for  the  use  of  Bayesian  methods  available  at  the  present  time.  l*rioi 
information  for  a  Bayesian  solution  might  include  student  mastery  SkCires 


ERIC 


125 


40 


on  other  skills  covered  on  the  test  or  student  performance  on  skills 
measured  on  previous  tests*     (In  the  case  of  posttestlng,  pretest 
Information  could  be  used  as  the  prior*)    Also,  just  as  data  from 
other  examinees  can  improve  the  precision  of  estimation  of  achievement 
In  a  norm-referenced  testing  situation  for  an  lndl\  Ldual  (Lord  and 
Noviclc,  1968),  so  perhaps  the  same  can  be  done  with  criterion-^ 
referenced  measurement  problems* 

Within  many  objective-based  programs  the  strategy  of  branched  testing 
would  seem  to  be  an  appropriate  technique,  at  least  In  situations  where 
the  objectives  In  a  content  area  can  be  arranged  Into  hierarchical 
sequences*    Some  of  the  practical  problems  have  been  resolved  In  the 
Pittsburgh  IPI  Program  so  that  the  technique  can  now  be  used  on  a  limited 
basis*    Nevertheless,  many  problems  remain  before  adoption  should  or  can 
proceed  with  other  programs*    For  example.  It  would  be  necessary  to  develop 
a  non-automated  modified  version  of  branched  testing  for  schools  without 
computers*    Also,  we  need  to  know  much  more  aboct  starting  places,  step 
slzest  storplng  rules,  etc*,  before  we  can  effectively  use  branched 
testing  in  an  instructional  setting* 
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1-  Introduction 

If  you  are  lucky>  and  certain  that  your  luck  will  hold>  you  should  read 
no  further.    Our  subject  is  decision  making  and  those  who  are  inherently  lucky 
will  have  no  need  to  attempt  to  take  advantage  of  a  logical  system  for  decision 
making.    However^  we  feel  that  logical  thought  has  been  successful  in  so  many 
areas  of  human  activity  that  it  deserves  a  chance  even  in  those  areas  where 
arbitrary  rules  or  intuition  seem  to  have  prevailed^  and  perhaps  been  moderately 
successful.    In  particular^  we  feel  that  the  Implementation  of  Individually 
Prescribed  Instruction  can  be  given  a  greater  payoff  if  a  coherent  system  of 
decision  making  is  incorporated  into  the  instructional  sequence  to  provide  a 
supplement  to  the  experience -honed  judgment  of  the  classroom  teacher. 

In  the  currently  popular  language  of  systems  engineering^  the  decision 
making  process  might  be  viewed  as  a  black  box.    The  black  box  contains  an 
input  hole  for  prior  information  about  the  environment  in  which  the  decision 
will  be  made  and  evaluated^  a  second  hole  for  new  experimental  results  designed 

^he  research  reported  herein  was  performed  pursuant  to  Grant  No.  OEG- 
0-72-0711  with  the  Office  of  Education^  U.  S.  Deprr'ment  of  Health,  Education, 
and  Welfare,  Melvin  R.  Novick,  Principal  Investigator*    Contractors  under- 
taking such  projects  under  Government  sponsorship  are  encouraged  to  express 
freely  their  professional  judgment  in  the  conduct  of  the  project.    Points  of 
view  or  opinions  stated  do  not,  therefore,  necessarily  represent  official 
Office  of  Education  position  or  policy*    We  are  grateful  to  Nancy  S.  Peterson 
for  carefully  reading  and  correcting  this  manuscript. 


2 


explicitly  to  cast  more  light  on  the  uncertain  environment,  and  finally  a  hole 
for  the  decision  maker^s  preferences  for  the  possi'i/^e  consequences  of  decisions. 
The  output  of  this  black  box  is  a  decision  or  action  that  will  ultimately 
lead  to  a  distr'^^'tlon  of  consequences  that  will,  hopefully,  be  high  on  the 
decision  maker's  preference  scale^    Our  business  will  be  to  get  enough 
of  a  look  at  what  goes  on  inside  the  black  box  so  that  we  can  see  how  the 
teacher  and  student  can  use  prior  information,  experimental  results,  and 
preferences  in  a  reasonable  way  to  generate  decisions  having  favorable 
consequences. 

Building  an  all'-purpose  black  box  for  decision  making  has  been  a  major 
project  in  recent  years.    Contributors  to  the  project  have  included  economists, 
mathematicians,  philosophers,  psychologists,  and  statisticians.    It  would 
take  a  major  treatise  to  adequately  trace  this  project*    We  must,  therefore, 
limit  our  survey  to  the  major  references  in  psychology  and  education  relevant 
to  our  task*    Those  elements  of  decision  analysis  which  require  that  existing 
information  about  the  process  under  study  be  quantified  in  order  that  new 
experimental  information  may  be  coherently  combined  with  the  existing 
information,  have  already  been  persuasively  presented  in  the  literature  of 
psychology  and  education  [Edwards*  Lindman,  and  Savage  (1963) However,  the 
requirement  that  preferences  for  various  possible  distributions  of  consequences 
be  formulated  coherently  and  expressed  as  a  numerical  valued  utility  function, 
has  not  been  emphasized  In  the  literature  of  education  with  the  singular 
excep*'ion  of  the  prophetic  text.  Psychological  Tests  and  Personnel  Decisions, 
by  Cronbach  and  Gleser  (1957,  1965)* 

Many  decision  makers  in  education  may  feel  that  business  managers*  with 
the  market  at  hand  to  evaluate  the  outcomcj  associated  with  their  decisions, 
are  In  a  more  favorable  position  to  make  value  comparisons  among  the 
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distributions  of  consequences  ^han  they  are.    For  example^  the  values  attached 
to  the  consequences  of  a  decision  to  Implement  a  specific  education  program^ 
or  to  advance  or  retain  a  particular  student  at  c  given  3*i;el>  seem  to  them 
to  defy  simple  quantification.      Such  decisions  appear  to  have  so  many 
possible  ramifications  that  to  formulate  preferences  with  any  degree  of 
consistency  Is  simply  Impossible* 

Nevertheless^  It  Is  our  position  that  most  data-collecting  activities  In 
education  are  designed  to  Influence  decisions.    In  fact>  to  simply  collect 
data  without  the  objective  of  ultimately  modifying  a  cov.'^e  of  action^  would 
Seem  wasteful.    We  further  assert  that  despite  all  the  perplexities^  decisions 
are  regularly  made  In  education  based  on  an  Informal  Qilxture  of  recently 
collected  Information^  prior  Information^  and  the  preferences  of  the  decision 
maker*    If  these  'leclslons  are  to  be  rational  In  the  sense  that  they  are 
derived  from  a  logical  program  for  decision  making  that  provides  for  the  Input 
of  prior  Information^  new  data»  and  preferences^  we  claim  that  decision  theory 
Is  required*    There  Is  no  magic  In  the  formal  structure  of  decision  theory. 
The  theory  contributes  only  what  mathematics  does  to  any  problem;  an  orderly^ 
systematic^  and  precise  framework  for  formulating  a  problem^  plus  the  economy 
of  mathematical  reasoning  In  tracing  the  consequences  of  the  formulation* 

The  main  difficulty  In  Implementing  decision  theory  arises  from  the 
necessity  to  quantify  basically  subjective  or  personallstlc  quantities^  and 
this  difficulty  is  real.    However^  we  believe  that  within  the  framework  of 
some  of  the  newer  and  very  highly  structured  modes  of  instruction^  it  is 
possible  to  provide  relatively  simple  yet  conclusively  meaningful  methods  of 
decision  analysis.    The  decision  machinery  that  we  shall  build  will  be 
appropriate  for  a  wide  band  of  decision  problems.    However^  we  will  consistently 
illustrate  the  ideas  with  examples  from  individualized  instructional  procf^nres. 
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2*    Fundamental  Ideas  and  Notation 

The  main  purpose  of  this  section  is  to  Illustrate  the  rudiments  of 
decision  analysis  with  a  variety  of  straightforward  examples*    Thus^  we 
begin  Section  2*1  with  a  rather  trivial  example*    In  later  sections^  this 
example  will  be  modified  and  extended  to  make  it  both  more  realistic  and 
interesting* 

2*1   The  Basic  Example 

The  environment  of  our  example  is  very  simple*    A  student  has  either 
mastered  or  not  mastered  the  topics  in  the  current  unit  of  his  individually 
prescribed  instructional  program*    The  state  of  the  student  being  a 
aonmaster  will  be  denoted  by  6    and  the  state  of  his  being  a  master  will  be 
denoted  by  6^*    Two  actions  are  open  to  the  decision  maker,  who  could  be 
the  teacher  or  even,  iu  some  arrangements,  the  student  himself.    The  decision 
may  be  to  retain  the  student  for  additional  work  at  the  present  level  o£ 
his  prescribed  program*    This  "retain"  decision  will  be  denoted  by  dy  The 
complement  of  this  decision  is  to  advance  the  student  to  the  next  unit* 
This  decision  will  be  denoted  by  d^* 

We  now  identify  the  three  basic  inputs  into  the  decision  analysis. 

(a)    Prior  information*    In  Individually  Prescribed  Iiii^truction,  a 
student  begins  a  unit  of  instruction  only  when  he  is  d&emed  to  be  prepared 
for  that  unit*    For  this  reason,  the  variation  in  v-^sttest  results  tend  tc  be 
relatively  small»  except  to  the  extent  that  they  are  due  to  sampling  variation. 
Therefore,  the  main  input  of  prior  Information  will  involve  our  beliefs  about 
the  relative  success  of  the  instr»:ctional  unit  with  qualified  entrants* 
Thus,  on  the  basis  of  an  examination  of  the  success  which  other  students 
have  had  on  this  training  unit,  and  before  administering  a  test  at  the  end  of 
the  present  unit,  the  decision  maker  assigns  prior  probabilities  to  the 
two  possible  states* 
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Table  1 
State  of  the  Student 

e    (nonmaster)  6  (master) 

Prior  ^  ^ 

Probability        f)(e)  ,4  *6 

This  Information  Is  relevant  to  the  decision  under  consideration  and  to 
Ignore  It  is  to  waste  useful  facts^ 

The  symbol  p(9)  denotes  the  prior  probability  mass  function*  This 
symbol  will  continue  to  play  the  same  role  throughout  our  discussion* 

(b)    Experimental  plan.    The  decision  maker  may  ask  the  student  one 
test  question  to  learn  more  about  which  state  prevails.    The  result  of  this 
short  test  will  be  denoted  by  X  .    If  the  student  answers  correctly,  X  =  1> 
and  if  he  answers  incorrectly^  X  =  0  .    The  probability  assignments  to  these 
two  outcom<iS>  if  the  mastery  state  is  known>  is  given  In  the  following  short 
table  where  p(x[e)  =  PrIX  -  x|e]  is  the  probability  mass  function  of  correct 
(x  *  1)  and  incorrect  (x  *  0)  responses  given  that  the  mastery    evel  is 
known  to  be  6  * 

Table  2 
p(x|e) 

X  =  0  (wrong)  x  =  1  (right) 

(nonmaster)  .8  .2 

62    (master)  .2  *8 

This  table  states  that  If  a  person  is  a  master^  then  the  probability  that  he 
will  give  a  correct  answer  is  *8  and  the  probability  that  he  will  give  a 
wrong  response  is  *2.    If  he  is  a  nonmaster^  the  probabilities  are  reversed* 
Note  that  those  are  probabilities  of  experimental  outcomes  given  the  true 
state  of  the  student  (^aster^  nonmaster)* 
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(c)    Preferences*    The  decision  maker  can  make  one  of  the  two  types  of 
errors*    If  he  retains  the  student  at  the  current  level  when,  In  fact,  the 
student  Is  a  master,  the  student  will  probably  repeat  the  current  unit  with 
only  minimal  gain*    On  the  other  hand*  if  the  student  is  advanced  when  he  has 
not  mastered  the  topics  on  the  current  level,  ultimately  he  may  have  to  repeat 
both  the  current  level  and  the  one  to  which  he  had  prematurely  been  advanced* 
With  these  facts  in  mind,  the  decision  maker  designates  the  nonnegatlve  loss 
function  L(d,  6)  defined  in  Table  3* 


In  specifying  a  loss  function  of  this  type,  the  decision  maker  assumes  that 
no  less  occurs  if  a  correct  decision  is  made  and  that  the  loss  associated 
with  advancing  a  nonmaater  is  twice  that  associated  with  retaining  a  master* 
We  do  not  suggest  that  this  simple  loss  function  is  appropriate  in  all  or 
even  a^y  situations*    However,  the  simple  assumptions  seem  realistic  enough 
to  maintain  our  attention  for  a  while* 

Three  reasonable  decision  rules  for  selecting  decisions  d^  or  d2,  after 
observing  the  test  score,  are  possible* 


Table  3 


L(d,  e) 


6  (nonmaster) 


(master) 


(retain) 
d-  (advance) 


Table  4 


Rule 


X  •  0  (wrong) 


X  »  1  (right) 


6j^(x) 

63  (X) 
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In  this  trivial  example^  decision  rules  and  ^jCx)  really  tell  us  to 

ignore  prior  information  and  the  current  test  result;  in  the  case  of 
to  take  action  d^^  (retain)  invariably,  and  in  the  case  of  to  stick 

with  action  d^  (advance)*    Nevertheless,  each  is  a  serious  candidate 
because  certain  loss  structures  and  prior  probability  distributions  could 
make  one  of  them  the  preferred  decision  rule* 

There  are  two  ways  that  the  analyses  carried  on  within  the  black  box 
labeled  ^'decision  process**  have  historically  been  organized*    The  first, 
which  is  called  normal  form  analysis^  involves  a  three  step  process* 
(1)  Compute  the  average  or  expected  loss  for  each  crdered  pair  (5^,  e^) 
composed  of  a  decision  rule  6^  and  a  state  parameter  value  6^  *  This 
averaging  is  performed  with  respect  to  the  probability  distribution  of 
possible  experimental  outcomes,  p(x|dj)* 

The  expected  value  that  emerges  from  this  computation  is  called  a  risk 

function  and  it  is  denoted  by  ^(2^*  6^)*    As  the  notation  suggests,  it  is  a 

function  of  both  the  decision  rule  6,  and  the  state  parameter  6,  *  In 

i  -  J 

symbols,  we  have 

R(6^.  Oj)  =  ZL[5^(x),  e^]p(x|6^) 

where  i  indexes  the  various  decision  rules  and  j  identifies  the  various  values 

of  the  state  parameter  6  *    Once  we  have  the  risk  function,  we  have  the 

expected  loss  for  each  possible  (<S^>  6^)  pair,  and  can  possibly  decide  that 

a  decision  rule  is  good  (bad)  if  its  risk  is  small  (large)  for  all  values 

of  e    *    (2)    This  kind  of  analysis,  however,  is  typically  inconclusive 
j 

so  we  must  compute  the  average  or  expected  value  of  the  risk  function  for 
each  decision  rule*    This  averaging  is  done  with  respect  to  the  prior 
probability  distribution  of  the  state  parameter  p(e)  *  This 
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expected  value,  is  called  the  Bayes  risk  function  and  will"  be  denoted  by 

p)>  where  the  symbol  p  Is  Insetted  to  stress  that  the  Bayes  risk  depends 
on  the  prior  probabdllty  distribution.    This  recipe  may  be  expressed  in 
symbols  as 

r(6  ,  p)  »  ZR(6  ,  e  )p(9  )  . 

The  effect  of  this  averaging  Is  to  weight  the  effect  of  a  decision  rule 
highly  for  those  values  of       that  we  thinks  aprlorl^  are  highly  probable 
and>  hence>  Important  to  consider*    (3)  With  this  computation  completed  for 
each  decision  rule»  we  select  and  subsequently  use  the  decision  rule  that  has 
the  smallest  Bayes  risk. 

Now  let  U6  carry  out  this  three  step  normal  form  analy:;ls  using  the 
losses  and  prior  probabilities  from  our  example.    Firsts  we  will  compute 
the  risk  of  decision  rule      when  the  true  state  of  the  person  Is  . 
Recall  that  with  6^^  we  retain  the  student  If  x  >  0  and  advance  him  if 
X  *  1  .    The  risk  Is 

R(62»  Qj)  »        Ax),  ej^)p(x|ej^) 

X 

l.e*^  the  risk  of  using  decision  rule  6^  when  the  true  state  Is  9^  Is  the 
simple  average  or  expected  loss  for  the  (62*  d^)  pair.    The  expectation  Is 
performed  with  respect  to  the  probability  distribution  p(x|9^)  of  the  two 
possible  test  scores  x  ■  0  and  x  *  1»  when  it  le  given  that  rlie  student  is 
a  nonmaster  (l^e*^  d  ■>  6^)*    Let  us  evaluate  this  risk.    From  Table  A,  we 
know  that  fijt*')  *  ^2^^^  "       *  Thus» 

RCfij*         -  t-Cd^*  9i)p(0|9j^)  +  L(d2,  9j^)p(l|e^)  . 
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From  Table  3>  we  have  Lid^^  0  and  LCd^*         =  2;  from  Table  2>  we 

have  p(olej^)  =  .8  and  p(1|Qj^)  =  .2.    Thus>  using  the  symbol  *  to  indicate 
multiplication^  we  have 

e^)  -  0  *  .8  +  2  *  .2  -  .4  , 

i.e.,  when  the  true  situation  is        decision  rule  ^2        ^  ^^^^  (expected 
or  average  loss)  of  .4.    Similar  computations  have  been  made  for  the 
remaining  i&^f  e_j)  pairs.    The  computations  have  been  suinmari^ed  in  Table  5, 


Table  5 


'  i:l[<s^(x)» 

X 

e^lp(xle.) 

state  Result 

P(x|e) 

<S^(x) 

<S3(x) 

e^^         X  =  0 

.8 

0  A  .8 

0  A  .8 

2  *  .8 

X  "  1 

.2 

0  *  .2 

2  *  .2 

2  *  .2 

0 

.4 

2.0 

^2          X  =  0 

.2 

1  *  .2 

1  *  .2 

0  *  .2 

X  »  1 

.8 

1  *  .8 

0  *  .8 

0  *  .8 

R(fi^»  e^) 

1.0 

.2 

0 

If  we  knew  that       (noiimster)  prevailed^  decision  rule  ^j^(^)  which 
always  retains  the  student  would  minimize  the  risk  function.    It  has  an 
expected  loss  of  zero  which  is  as  good  as  one  can  do.    If  we  knew  that  ^2 
(master)  prevailed,  decision  rule  ^^f^)  which  always  advances  the  student 
would  minimize  the  risk  function.    Again  this  rule  would  have  no  risk. 
||^[^(^)f  course^  if  we  knew  which  state  p?fevailed  there  would  be  no  need  to  apply 
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decision  analysis  to  the  problem.    At  this  point,  none  of  the  three  decision 
rules  are  judged  to  be  uniformly  superior,  i.e.,  superior  for  every  state  of 
nature.     If  one  were,  we  would  certainly  adopt  it,  but  none  is,  so  we  must 
find  some  way  of  choosing  the  best  decision  rule.    We,  therefore,  move  to 
Step  (2)  in  normal  form  analysis  before  identifying  the  winner.  The 
computation  of  the  Bayes  risk  must  now  be  made  for  each  of  the  three  decision 
rules.    The  idea  is  simply  this.    We  do  not  know  the  true  state  of  the  person, 
but  we  do  have  a  prior  opinion  concerning  the  true  state.    Therefore,  it  makes 
sense  to  average  the  risk  for  each  decision  rule  with  respect  to  our  prior 
opinions,  in  effect,  to  put  more  weight  on  those  values  of  6  that  seem  more 
probable  to  us.    Por  example,  to  compute  the  Bayes  risk  r(£,  p)  for  ^2  (^he 
average  risk  with  respect  to  our  prior  probabilities  for  9^^  and  d^) ,  we 
compute  the  following: 

Substituting  values  from  Table  5  and  Table  1,  we  have 

(.A  *  .A)  +  (.2  *  .6)  =  .28  . 

Tlie  computations  of  the  Bayes  risk  for  6^  and  6^  are  also  easily  made  and 
are  given,  together  with  those  for        in  Table  6. 


Table  6 


P(e) 


.A 


.6 


r(6j^»  P) 


R(fi^»  e)p(e) 


0  *  .A 


1  *  .6 


.6 


.A  *  .A 


.2  *  .6 


.28 


2  *  .A 


0  *  .6 


.80 
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The  final  step  in  normal  form  analysis  directs  us  to  select  action 
rule  6„  b  ecause  It  hcis  the  smallest  Bayes  risk*    Thus,  we  have  stated  and 
exhibited  in  detail  a  precise  and  coherent  procedure  for  decision-making 
in  the  presence  of  uncertainty*    We  have  further  demonstrated  a  simple 
application  in  the  context  of  Individually  Prescribed  Instruction*    To  see 
that  the  choice  of  decision  rule  really  depends  on  prior  probabilities, 
the  reader  should  redo  step  (3)  in  the  analysis  first  with  Pi^^)  ^  -^t 
pCe^)  -  *9,  and  then  with  p(8j^)  =  *9,  pCe^)  =  *1* 

A  second  way  of  organizing  the  analysis  within  the  decision  process 
black  box  is  called  extensive  form  analysis*    Since  this  type  of  analysis 
has  some  computational  advantages  over  the  normal  form,  our  subsequent 
illustrations  will,  with  one  exception,  employ  extensive  form  analysis* 

Extensive  form  analysis  also  involves  a  three  step  process* 

(1)    Determine  the  posterior  probability  distribution  of  the  state  parameter 

6,  given  the  experimental  result  x  *    That  is,  we  must  determine 

p(6|x)  =  p(jc[e)p(9)/p(x),  where  p(x)  =  Ip(xje)p(e)  is  the  unconditional 

9 

probability  mass  function  of  X  .    The  posterior  distribution  p(8tx)  summarizes 

our  knowledge  and  beliefs  about  9,  incorporating  both  our  prior  beliefs  and 

the  sample  information*    (2)    Compute  the  expected  value  of  the  loss  function 

for  each  decision  rule  with  respect  to  the  posterior  distribution  of  9  * 

That  is,  we  must  compute  11(6.,  8)p(8|x)  for  each  decision  rule*     (3)  Select 

6 

the  decision  that  will  yield  the  smallest  posterior  expected  loss  In  Step  (2)* 

The  advantage  of  extensive  form  analysis  arises  in  Step  (2)  and  Is  a 
bit  hard  to  appreciate  when  expressed  only  in  words.    The  heart  of  the 
matter  is  that  Step  (2)  does  not  have  to  be  carried  out  for  every  possible 
value  of  X  *    If  we  adopt  this  system,  we  can  wait  and  perform  Step  (2)  only 
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tor  the  result  x  that  Is  actually  observed.  Once  x  is  observed,  the  decision 
rule  specifies  the  decision  d^  to  be  made>  and  the  losses  under  each  state 
e  can  be  taken  immediately  from  Table  3* 

The  amazing  thitift  is  that  under  very  general  conditions^  normal  and 
extensive  form  analygie  will  lead  to  the  same  decision.    This  point  will  be 
illustrated  in  our  example*    Later,  a  mathematical  argument  will  be  presented 
for  those  who  can  only  be  persuaded  by  such  demonstrations  (Raiffa  and 
Schlaifer,  1961,  p*  15). 

The  first  step  in  extensive  form  analysis  requires  us  to  determine  the 
form  of  the  posterior  probability  distribution  of  6  *    For  our  example, 
we  will  do  more  and  actually  exhibit  the  two  possible  posterior  distributions* 
But  first,  we  need  to  exhibit  the  joint  probability  distribution  of  X  and  0, 
and  the  marginal  distribution  of  X  *    These  probabilities  are  given  in  Table  7 
where  the  entries  in  the  body  of  the  table  are  the  joint  probabilities » 
and  the  entries  in  the  margins  are  the  marginal  probabilities*    In  computing 
Table  7,  we  have  used  P(6)  from  Table  1  and  p(x]6)  from  Table  2* 

Table  7 
p(x|e)p(0)  '  p(x,  e) 

X  «=  0  X  =  1  P(0) 

.40 
*60 

p(x)  *44  .56 

Then  the  posterior  probability  distribution  of  6  for  given  values  of  x 
is  given  by  Bayes  Theorem  p(Olx)  ■  p(x>  0)/p(x)  .    These  conditional  prob- 
abilities for  3  given  x  are  summarized  in  Table  8.    Note  that  the  conditionaJ 
distributton  of  0  given  x  ■  0  is  very  different  from  that  given  x     1  * 


.32 

.08 

.12 

.48 

146 


13 


Table  8 
p(6|x) 


X  «  0 

X  =  1 

^1 

.73 

.27 

.86 

1.00 

1.00 

The  second  step  in  extensive  form  analysis  calls  for  the  computation 
of  the  average  or  expected  value  of  the  loss  function  for  each  decision 
rule  with  respect  to  the  posterior  distribution  of  6  [l»e*,  we  must  compute 

Qj)p(ej|x)]*    We  will  carry  out  this  computation  for  each  of  the  three 
decision  rules  specified  in  Table  4*    Suppose  x  =  0*    Then  from  Table  8,  we 
see  that  p(6j^|x  -  0)      *73  and  ?(^2^^  =  0)  ^  *27*    If  we  adopt  6^,  then  we 
shall  make  decision       when  x  =  0  (see  Table  4)*    Thus,  If  9  =  6^^,  our  loss 
will  be  2ero,  and  If  9  =  e^*  our  loss  will  be  one  (see  Table  3)*  Therefore^ 
our  average  or  expected  loss  given  x  =  0  Is 

£.a[5j^(0),  ejlx  =  0)  =  L[5^(0),  9j^lp(eil==  =  0) 

«  (0  ^  .73)  +  (1  *  *27)  =  *27 
as  given  In  the  first  column  of  Table  9* 
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Table  9 

{L[6^(x),  eilx} 


X 

-  0 

X  -  1 

*1 

h 

^3 

\ 

^2 

0 

*  .73 

0 

*  .73 

2  *  .73 

0 

*  .14 

2  *  .14 

2  *  .14 

1 

*  .27 

1 

*  .27 

0  *  .27 

1 

*  .86 

0  *  .86 

0  *  .86 

.27 

.27 

1.46 

.86 

.28 

.28 

In  this  table^  aimilar  comptitationa  are  made  for  each  of  the.  three  decision 
rules  for  both  x  "  0  and  x  "  1.    Observe  that  decision  rule  &2^^^  produces 
the  smallest  expected  loss  for  each  value  of  x  and  may  be  judged  as  the 
optimum  decision  rule*    However^  if  x  »  0>  rule  6^  is  equally  as  good>  and 
if  X  =  1,  rule  6^  ^®  equally  as  good.    More  Importantly,  however,  thf.s 

observation  indicates  that  the  second  and  third  steps  can  ba  simplified 
still  further.    Note  that  if  X  »  0,  5j^(x)  and  Sj^^^  same  decision 

(dj^)  and,  consequently,  must  necessarily  have  the  same  expected  loss.  What 
this  emphasizes  is  that  once  ve  kaow  x»  we  are  really  Interested  only  In 
the  best  decj.aj.9p  ^2^*  decision  rule  (5^^,         5^)  ^^^^ 

will  lead  to  the  best  decision.    Consequently^  we  need  only  compute  the 
expected  or  average  valv^  of  the  loss  function  for  the  available  decisions. 
That  decifllon  (not  decision  rule)  with  the  smallest  posterior  loss  will 
then  be  selected.    This  is  done  in  Table  10. 
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Table 

10 

EL(d»  6)p(9|x) 
9 

=  0 

1 

•^l 

''l 

'2 

0  *  .73 

2  *  .73 

0  *  .14 

2 

*  .14 

1  *  .27 

0  *  .27 

1  *  .86 

0 

*  .66 

.27 

1.46 

.86 

.28 

Our  action  rule  is  the  same  as  Indicated  previously;  if  we  observe 
X  »  0>  we  will  take  action       (retain) >  and  if  we  observe  x  *  1>  we  will 
take  action  d^  (advance) «    This  i8»  In  effect^  the  same  as  adopting 
decision  rule  &2  *    ^  pointed  out  previously^  in  extensive  form  analysis^ 
only  that  half  of  Table  10  which  correspond.^  to  the  actual  result  observed 
(x  ■  0  or  X  =  1)  needs  to  be  calculated* 

So  far,  we  have  illustrated  two  forma  of  analysis  which  we  claim  will 
lead  to  identical  decisions.    These  forms  combine  prior  information, 
experimental  results,  and  preferences  using  the  loachinery  of  probability 
theory  to  trace  the  consequences  of  the  inputs.    The  example  that  helped 
to  illustrate  these  ideas  was  kept  at  a  trivial  level  so  that  the  arithmetic 
would  not  obscure  the  essence  of  the  process. 

2*2    The  Formal  Structure  of  Decision  Theory 

Although  mathematical  symbols  may  seem  forbidding,  they  are  an 
indispensable  tool  in  conveying  ideas  precisely.    Therefore,  both  for 
completeness  and  for  precision  we  will  retrace  the  key  ideas  of  decision 
analysis  relying  on  symbols  rather  than  numbers.    Although  integral  signs, 
/,  will  be  used,  we  remind  our  readers  that  this  is  merely  a  continuous 
analogue  of  the  summation  operator,  S  . 
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We  l4t  L[5(x)>  8)]  be  a  nonnegatlve  loss  function  defined  on 
t\\o  ti<dt  of  pairs  of  the  param^ater  values,  6  and  Q  .    The  symbol 

Lt;  ujed  to  emphasize  the  fact  tliat  the  decision  rulers  usuaJly 
depend  on  obstrV«d  values  of  the  random  variable  X;  Che  observation 
being  obtained  to  elicit  Information  about  6  .    In  some  contexts  it 
may  help  to  think  of  6  as  the  mean  of  some  probability  distribution, 
and  6(x)  ae  an  tstlMt*  of  6  .    Then  L[6(x),  6j  is  the  nonnegatlve 
numerical  value  of  the  lose  associated  with  using  6(x)  as  an  estimate 
the  irean,  when»  in  fact,  the  mean  is  0  * 

In  the  flr*t  stage  of  normal  form  analysis,  we  need  the  expected 
or  average  value  of  this  loss  function,  where  expectation  is  taken  with 
respect  to  the  coadltional  distribution  of  X  given  6    .    Tliis  expected 
value  was  ref<rr«d  to  as  the  risk  function  and  was  denoted  R(6,  Q)  * 
That  is 

R(6,  e)      /L[5(x),  O]p(x|0)d^;  . 

t'^or  this  integral,  the  symbol  p(xj6)  denotes  the  probability  density 
of  the  random  variable  X,  given  the  value  of  the  rotate  parameter  0  . 
If  X  is  a  dlacreta  random  variable,  p(x]6)  is  to  be  Interpreted  as  a 
probability  mass  function,  and  integration  is  then  replaced  by  summation. 
As  the  symbols  suggest,  the  risk  funcdon  is  similar  to  the  loss  function* 
The  major  differance  tests  on  the  fact  that  the  risk  function  no  longer 
dapandt  upon  tht  observed  value  of  our  experimental  variable  X;  X  has 
been  averagad  out«    Thle  fact  is  emphasized  by  the  absence  of  an  x  in 
our  notation  for  the  rl3k  function,  R(6,  6)  . 

If  one  adopts  a  Bayeslan  view  of  statistics,  and  one  it;  compelled 
to  embrace  this  view  if  he  accepts  any  of  several  comprehensive  axiom 
systems  for  decision  making,  it  becomes  necessary  to  quantify  previous 
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or  collateral  information  about  the  state  parameter  6^  which  we  summarize 
In  pC6),    In  the  second  stage  of  normal  form  analysis^  the  expected  or 
average  value  of  the  risk  function  from  stage  one  Is  then  calculated  with 
respect  to  this  prior  distribution^  pCe).    We  referred  to  this  expected 
risk  as  tha  Bayes  risk  associated  with  6>  and  denoted  lt>  rC6>  p)  .    And  so 
we  have> 

r(6,  p)  =  fR{s,  e)pce)de 

=  /{/L[5(x),  e]pCx|e)dx}p(e)de  . 

Xhen»  the  third  and  final  stage  of  the  normal  form  analysis  consisted  of 
selecting  the  decision  rule  which  minimizes  the  Bayes  risk. 

As  was  pointed  out  above,  extensive  form  analysis  follows  a  slightly 
different  route^  but  under  rather  general  conditions,  leads  one  to  the 
same  decision.    In  extensive  form  analysis >  we  begin  by  evaluating  the 
posterior  distribution  of  6 

pCe)pCx|e) 


P(e|x) 


and  then  determine  the  expected  loss  with  respect  to  this  posterior 
distribution.    In  the  continuous  case>  this  expected  or  average  loss  can 
be  represented  by  the  integral^ 

I  a[6(x),  el)  =  /L[6Cx),  e]p(elx)de  . 
|x 

Naturally^  in  the  discrete  case>  we  merely  replace  the  integration  operation 
with  that  of  summation*    The  decision  making  criterion  in  extensive  form 
analysis  is  then  to  choose  that  decision  rule  6(x)>  which  minimizes  this 
expected  loss*    The  reader  should  take  special  note  at  this  pointy  that 
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although  this  exp«ctad  losi  depends  uPon  the  random  variable  X»  It  needs  to 
be  evaluated  only  for  that  value  x  which  Is  actually  obtained* 

In  order  to  make  clear  the  relationship  between  normal  form  and 
extensive  form  analyses^  let  u«  compare  the  decision  criteria  In  the  two 
cases*    Under  normal  fona  analysis  we  are  to  choose  the  decision  which  minimi 
the  Bayes  rlsk> 

r(fi,  p)  -  /{/L[6(x),  6]p(x|6)dx}p(0)d6  • 

Now>  if  these  integrals  are  suitably  well  behaved^  we  may  interchange  the 
order  of  integration  and  so  write  the  Bayes  risk  in  the  fortn» 

r(fi,  p)  -  /{/Lt5(x),  6]p(x|6)p(6)d0ldx  * 

Since  by  Bayes  theorem  p(x|6)p(6)      p(dix)p?x),  we  may  rewrite  this  last 
equation  in  the  form 

r(fi,  p)  -  /{/L[5(x),  6]p(6|x)d6}p(x)dx    *  (2*1) 

The  observant  reader  will  of  course  already  have  recognized  the  integral 
in  brackets  as  the  expected  loss  which  must  be  minimiztd  under  the  extensive 
form  approach*    This  equation  illuetrates  the  crucial  difference  between 
the  two  approaches*    Extanaive  form  analysis  chooses  the  decision  rule 
which  minimizee  the  *xpMted  loss  for  the  particular  value  of  x  observed* 
In  contrast >  normal  form  analysis  chooses  the  decision  rule  which  minimizes 
the  average  of  those  expected  losses  for  all  possible  values  of  x*  Clearly, 
if  one  particular  decision  rule  minimizes  the  expected  loss  criterion  of 
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extensive  form  analysis  for  every       then  the  average  of  those  expected 
losses  under  that  decision  rule  must  also  be  a  minimum*    In  this  case,  it  is 
clear  that  the  decision  taken  under  extensive  form  analysis  will  coincide 
precisely  with  that  taken  under  normal  form  analysis*    However,  in  those 
Instances  where  extensive  form  analysis  apparently  leads  to  different 
decision  rules  depending  upon  the  value  of  x  observed,  the  equivalence  of 
the  two  approaches  may  not  be  obvious* 

Although  the  possible  non-aquivalence  of  these  two  approaches  may 
seed  to  pose  a  dilemma  for  users  of  decision  theory,  in  fact,  it  is  a 
non-problem  which  has  been  set  merely  for  the  pedagogical  purpose  of 
underscoring  some  fundamental  differences  in  the  two  approaches*  Consider, 
if  you  will,  the  following*    Since  we  are  admitting  for  consideration  all 
reasonable  decision  rules,  we  must  allow  that  rule  £  (x)  which,  for  each 
X,  minimizes  the  expected  loss  criterion  of  extensive  form  analysis* 
Since  £  (x)  will  be  selected  by  the  extensive  form  approach  irrespective 
of  the  value  of  x  obtained,  it  will  also  be  chosen  under  the  normal 
form  procedure*    How  do  we  construct  6  (x)?    We  do  this  in  a  straight- 
forward operational  manner:    We  use  extensive  form  analysis  for 
the  X  obtained  and  choose  that  decision  (not  decision  rule)  which  minimizes 
the  expected  loss* 

For  a  concrete  example  of  the  relationship  between  normal  form  and 
extensive  form  analysis,  consider  Table  11  which  summarizes  the  expected 
losses  under  extensive  form  analysis  for  the  three  decision  rules*  This 
table  is  merely  a  modified  version  of  Table  9* 
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Table  11 


{L[6,(x),  01) 


«2 

X  -  0 

,27 

.27 

1.46  ' 

X  -  1 

.86 

.28 

.28  i 

We  found  In  Table  7  that  the  marginal  distribution  of  X  Is  given  by 
pr(X  =  0)  =  .44  and  pr(X  *  1)  =  .56.    Thus,  as  inc'  cated  in  Equation  (2.1), 
the  Bayes  risk  associated  with  decision  6^  is  given  by  the  weighted  average 
of  the  entries  In  the  first  coluion  of  Table  11,  where  the  weights  are 
the  marginal  probabilities  of  X  .    And  so  we  have. 


r(6j^,  P)  »  (-27  *  .44)  +  (.86  *  .56)  =  .6 


as  we  saw  in  Table  6. 

2.3    Extensive  Form  Analysis  ylth  a  Continuous  Posterior 

Let  us  now  turn  to  an  example  of  extensive  form  analysis  which  uses 
a  continuous  model  density.    This  eicample  is  only  a  slight  modification 
of  that  used  previously;  the  primary  difference  being  that  we  now  assume 
that  both  the  state  parameter  6  and  the  random  variable  X  are  continuous. 
It  should  be  noted  that  this  example  is  merely  a  reformulation  of  one 
considered  by  Hambleton  and  Novlck  (1972).    As  before,  two  decisions  are 
open  to  the  decision  maker  who  is  guiding  a  student  through  an  ordered 
sequence  of  instructional  units.    At  the  end  of  each  unit,  the  decision 
makeri  based  on  hi*  knowledge  of  the  student ^s  past  performance,  the 
performance  of  similar  students^  and  current  te^^t  results*  must  decide 
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to  advance  Che  student  Co  Che  nexC  unic  In  Che  sequence  or  to  reCaln  him 
aC  the  presenC  level.    If  Che  decision  maker  knovs  Che  studenC*s  masCery 

level  he  would  be  willing  Co  advance  Che  sCudenC  if  9  >  0    and  Co 

o 

retain  him  if  0  <  0    »    Thus*  Che  number  9    is  a  cuCoff  (or  selecCion) 

O  '  O  N  *  / 

poinC  on  Che  masCery  scale  with  respect  to  Che  acCions  advance  or  reCain» 

In  selecCing  6^,  careful  consideration  of  Che  objectives  of  the 
Cralning  program^  and  previous  experience  wiCh  Che  Craining  and  evaluaCion 
materials  musC  prevail*    If  for  example,       is  InCended  merely  to  give 
at  least  an  even  chance  of  completing  the  nexC  lesson^  6^  mighC  be  seC 
equal  to  ChaC  level  of  functioning  which  has  hisCorically  had  a  50%  success 
rate  on  the  nexC  uniC»    If*  on  Che  other  hand*  Che  decision  maker  is  very 
concerned  about  Che  ill-effecCs  of  Che  frustraCion  of  a  poorly  prepared 
studenC  reading  advanced  material,  perhaps  6^  oughC  Co  be  somewhaC 
higher*    In  any  case,  once  6^  is  specified  for  Che  Cest,  prior  and  collaCeral 
InformaCion  abouc  Che  sCudenC  will  be  combined  wiCh  Che  tesC  result  (x) 
for  the  purpose  of  estimating  6  , 

Assume  ChaC  for  Chis  two-action  (advance  or  retain)  problem*  the  decision 
maker  specifies  a  threshold  (or  step)  loss  function  which  can  be  described 
by  the  following  table:    (Compare  with  Table  3) 

Table  12 


L(d^*  e) 


(retain) 
d^  (advance) 


e  <  e 


B  >  & 

o 
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where  b  and  c  are  both  nonnegative«    In  the  literature^  losses  associated 
with  falling  into  the  lower  left  cell  of  this  table  are  frequently  referred 
to  as  arising  from  "f alse"positives'^  since  the  decision  maker  has  wrongly 
presumed  that  the  parameter  6  lies  in  the  region  which  has  positive  ethical 
value  (i*e*>  has  wrongly  presumed  that  9  >_  9^)*    Similarly^  losses  associated 
with  falling  in  the  upper  right  cell  are  commonly  referred  to  as  arising 
from  '^false-negatives*'*    The  nonnegative  numbers^  b  and  c>  reflect  the  cost 
of  making  these  two  types  of  errors*    Because  the  decision  envisaged  in  this 
example  is  rather  locals  affecting  only  one  step  in  a  program  which  is  only 
a  small  part  of  the  student's  total  learning  experience >  a  massive  effort 
to  determine  b  and  c  exactly  would  seem  inappropriate*    In  some  cases^  it 
might  seem  reasonable  to  assume  that  c/b  «  2^  if  for  example^  a  false 
positive  results  in  repeating  two  steps  in  the  se^.^uce^  as  compared  with 
the  repetition  of  only  one  for  a  false  negative* 

Following  the  general  scheme  for  extensive  form  analysis  outlined 
earlier »  our  goal  is  to  determine  the  action  which  will  minimize  the 
expected  or  average  posterior  loss 

A.(d^,  e)p(e|x)d6  * 

This  integral  is  equal  to  b{Pr[9  >_  if  i  =  1  and  is  equal  to 

c{Pr[0  <  if  i  =  2*    Therefore^  we  may  minimize  this  expected  loss 

by  making  decision  dj^  for  those  values  of  x  such  that 

b{Pr[9  >  e  |x]}  <  c{Pr[e  <  e  |x]} 
—   o  o 

and  by  making  decision  d^  for  those  values  of  x  vhere 

b{Pr[0  >_  0  \x])  >  c{Pr[0  <  6  |  x]}  . 
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Although  such  a  situation  is  unlikely  in  practice^  when  the  two  possible 
values  of  the  integral  are  equals  we  will  be  indifferent  about  which  decision 
to  take. 

In  many  applications  of  this  type>  the  range  of  test  scores  may  be 
cut  into  a  decision       region  and  a  decision  d^  region  by  considering  the 
posterior  distribution  of  0[i*e*>  p(e|x))  as  a  function  of  x  *    Doing  this> 
we  see  that  the  critical  point  dividing  the  two  regions  can  be  represented 
by  that  point       in  the  set  of  possible  test  results  such  that 

c{Pr[e  <  0  |x  ]}  =  b{Pr[e  >  6  Ix  ]} 
o'  o  —   o*  o 


or 


so  that 


c{Pr[e  <  e  jx  ]}  -  b{i  -  Pr[6  <  6  Ix  ]} 
o'  o  ^         o^  o 


— ^  -  Pr[e  <  0  Ix  1  (2.2) 

C  +  D  O '  O 

What  this  equation  says  is  that  if  we  consider  the  class  of  possible  posterior 
distributions  {p(e|x)}  to  be  indexed  by  x>  and  if  we  can  find  that  member  of  this 
cla^s  which  is  identified  by  x  =  x^,  say,  such  that  PrfQ  <  ^^l^^l  "  ^^^^  + 
then  for  ail  x  <  x^,  we  will  choose  decision  d^^  and  for  all  x  >  x^,  we  will 
select  decision  d2  * 

We  will  illustrate  the  computation  of  the  cutting  score  x^  on  our 

observation  scale  with  an  example.    Suppose  that  the  test  score  X  has  a 

2 

normal  distribution  with  unknown  mean  6  and  known  variance  o    *  Further, 
suppose  that  all  existing  information  about  the  parameter  0,  which  measures 
the  mastery  level  of  a  certain  skill,  may  be  summarized  by  a  normal  distribution 
with  mean  x  and  variance  <|>  *    Then,  a  simple  application  of  Bayes  theorem 


2 

The  uniqueness  of  the  point  x^  satisfying  this  equation  is  presumed. 


15/ 
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yields  3  posterior  distribution  of  6  which  Is  normal  3nd  h3S  3  ine3n  of 

2  2  2  2 

(to    +  X(t)/(cr    +  ^)  and  3  V3rl3nce  of  o  +  <?  )•    Transforming  this 

posterior  distribution  on  6  Into  st3nd3rd  form^  ve  see  th3t 


Prte  <  e  |x  ]  «  Pr 


z  < 


Tcr  +  X  4> 
e   ^ 


where  z  h3S  3  normal  distribution  with  TQe3n  zero  3nd  V3riance  one* 

Therefore,  the  cutting  score  x  on  the  observation  sc3le  may  be  determined 

o 

by  finding  that  point       In  the  standard  norm3l  distribution  which  h3S 

percentile  rank  106[b/(c  +  b)],  and  then  solving  the  equation 


TO    +  X  4> 

A  el 


for  X    •  Thus* 
0  * 


=0  -^'^-^o ^"')l'^>-T 


(2-3) 


In  order  to  convey  some  feeling  for  how  this  loss  structure  and  the 

norm3l  data  and  normal  prior  distributions  interact  to  produce  cutting 

scores,  Table  13  has  been  provided*    In  this  table,  the  desired  proficiency 

level  0^  «  75  and  the  prior  mesn  t  =  80* 

2 

As  we  would  expect  with  ^  and  cr    held  const3nt,  the  cutting  ijcore  x 

o 

Increases  with  c/b,  the  relative  loss  for  3  false  positive  as  comp3red  with 
a  f3lse  neg3tlve  error*    From  3n  intuitive  point  of  view,  this  relationship 
makes  sense*    As  c/b  Increases,  f3lse  positive  errors  become  relatively  more 
expensive  than  those  of  the  false  negative  variety^    Consequently «  our 
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Table  13 


C 

1  ^ 

Prior  <t  1  Data  a 

roscerior 
Variance 

roscerior  uuccxng 
Score  X 

Critical  Posterior 
Mean 

OU  O     T  X  9 
0 

0    +  <f 

I 

9 

9,0 

9.0 

4.5 

79  9ft 
/  Z  .  ZO 

I 

9 

9,0 

16.0 

5.8 

S7  Sft 

71  Q'^ 
/ 1  .  7  J 

I 

9 

25.0 

6.6 

ift  67 

71  71 

9 

16.0 

9.0 

5  8 

67 

71  Q'^ 

I 

9 

16.0 

16.0 

8.0 

fi^  7fi 

71  '^ft 

9 

16.0 

25.0 

9.8 

56  94 

71 

9 

25.0 

9.0 

6  6 

6ft  79 

71  71 

9 

25.0 

16.0 

9  8 

65  ?i 

71  CiCi 

9 

25.0 

25.0 

7*1    A  7 

7 

9.0 

9.0 

L  5 

7  77 

/J  .37 

3 

7 

9.0 

16.0 

^ .  o 

0  Z  .01 

71   7  A 

3 

7 

9.0 

25.0 

6  6 

/  J  .  0  J 

3 

7 

16.0 

9.0 

5  ft 

1(\    9  9 

7'^  7A 

3 

7 

16.0 

16.0 

ft  0 

0  /  .y  J 

71  "19 
/  J  .  JZ 

3 

7 

16.0 

25.0 

Q  A 

*i9  QQ 

OZ  .77 

71  Irfi 

/J.  Jo 

3 

7 

25.0 

9.0 

u  .  u 

71 

/ 1 .  JO 

71  ^^5 

3 

7 

23.0 

16.0 

Q  51 

6Q  1 1 

U7  .  ±± 

71  16 
/  J  .  JO 

3 

7 

Z3  .0 

^  C  A 

25.0 

±^  .  ^ 

66  9Q 

UU  .  ^7 

71  1A 

5 

5 

9.0 

n  A 

9.0 

n  .  *i 

7(1  CiCi 

7^  {\{\ 

5 

5 

y  .0 

T  ^  A 

16 .0 

S  ft 

66  11 

7^ 

5 

5 

y  .0 

O  C  A 

25 .0 

6  6 

6l  11 

U  X  .  X  X 

5 

5 

lo.O 

n  A 

9.0 

5  8 

72  19 

75  00 

5 

5 

16 .0 

16.0 

8  0 

70  00 

75 

5 

5 

16.0 

25.0 

67  1  Q 

U  /  .  X  7 

75  fin 

5 

5 

25.0 

9.0 

6  b 

7'^  90 

7^  CiCi 

5 

5 

25.0 

T  ^  A 

16.0 

9.8 

71  ftO 

75  00 

5 

5 

25 .0 

^  C  A 

25.0 

70  00 

75  00 

7 

3 

Q  f\ 

Q  f\ 

y  .u 

i  5 

79  9'^ 

76  1 1 

/  U  .  X  X 

7 

3 

9.0 

16.0 

5.8 

6Q  61 

U7  .  UX 

76  ?6 

7 

3 

9.0 

25.0 

6.6 

66  21 

76  15 

7 

3 

16.0 

9.0 

5  8 

74  16 

/  H  *  XU 

76  ?6 

7 

3 

16.0 

16.0 

8.0 

7?  Q7 

76  ift 
/  0  . 4  0 

7 

16.0 

25.0 

9.8 

71 

7^;  ^lA 
/  0  .  D  4 

7 

25.0 

9.0 

6  6 

7^  flA 

7*;  1^ 

/o .  JO 

7 

25.0 

16.0 

9  8 

7i  iQ 

76  6A 
/ 0  . Oh 

7 

25.0 

25.0 

12.5 

73.71 

76  ft6 

9 

9.0 

9.0 

75  43 

77  72 

9 

1 

9.0 

16.0 

5.8 

74.64 

78.07 

9 

9.0 

25.0 

6.6 

73.55 

78.29 

9 

16.0 

9.0 

5.8 

76.99 

78.07 

9 

16.0  . 

_16.0 

8.0 

77.24 

78.62 

9 

16.0 

25.0 

9.8 

77.43 

79.00 

9 

25.0 

9.0 

6.6 

77.68 

78.29 

9 

25.0 

16.0 

9.8 

78.36 

79.00 

9 

^  1 

25.0 

25.0 

12.5 

79.05 

79.53 
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decisi/>n  maker  decreases  his  chances  of  making  false  positive  errors  by 
Increasing  the  cutting  score,  . 

Since  the  mean  of  the  posterior  distributitn  of  6  is  a  linear  function 
of  K,  we  may  reformulate  the  question  of  the  critical  cutting  score  into 

2  2  2 

a  question  concerning  the  critical  posterior  mean  li^  =  (to    -f  x^<}>)/[4>a  /(a    -f  ^] 
where       Is  determined  by  Equation  (2*3)*    In  this  case,  we  solve  the 
equation 

e  -  p 

0  o 

for       -    And  analogous  to  Equation  (2*3)  above  for  k^,  we  have 

o       o  o 

In  preparing  Table  13,  we  also  calculated  values  of       which  appear 
in  the  last  column.    These  results  can  be  understood  by  referring  to  Figure  2-1 
Our  procedure  says  that  the  critical  posterior  distribution  of  e  must  be  such 
that  100[b/(c  +  b))  percent  of  the  probability  lies  below  6^  -    In  order 
to  maintain  this  constant  percentage,  as  the  posterior  variance  lncreas«iS, 
the  critical  posterior  mean  of  6  must  necessarily  decrease  for      ^      >  .5 
and  Increase  for      ^      <  *5*    That  is^  as  the  posterior  variance  of  6 
increases,  the  critical  posterior  mean  moves  ^way  from       .    From  an  intuitive 
point  of  view,  this  makes  sense*    It  Implies  that  as  a  decision  maker 
becomes  increasl^iSly  uncertain  about  the  posterior  mean  as  an  estimate  of 
6,  he  becomes  nore  cautlouS|  moving  his  critical  mean  in  the  direction  o£  the 
Hss  costly  errors* 

The  effects  of  changes  in  parameter  values  on       is  a  bit  more 
complicated  since  we  must  consider  not  only  the  posterior  variance^  but 
also  the  ratio  of  the  prior  variance  (<{>)  to  the  variance  of  sampling 
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^  Figure  2.1.    This  figure  Illustrates  the  necessary  change  In  the  critical 
posterior  mean  as  the  posterior  variance  Increases.    In  the  figure,  we  let 
b/(b  +  c)  «  .9  In  (1),  (111),  and  (v).    We  let  b/(b  +  c)  =  .1  in  (11),  (Iv) , 
and  (vl).    The  figure  Illustrates  that  for  b/(b  +  c)  >  .5,  the  critical 
posterior  mean  o£  O  must  decrease  with  Increasing  variance,  while  for 
b/(b  +  c)  <  .5  It  must  increase. 
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distribution  of  x  (i*e*,  o  ).  One  can  get  some  notion  of  what  is  going  on  by 
examining  the  definition  of  p  . 


TCF    +  X  <|> 

 o 

^o  °  — 2  

a   +  <)> 

Since       is  the  weighted  average  of  t  and  x  ,  for  given  u  ,  x    must  increase 
as  the  ratio  ^/o    increases*    This  relationship  is  clearly  indicated  in 
Table  13* 

At  this  point  let  ua  look  at  a  numerical  example  of  some  of  the  theory 
we  have  developed  in  the  last  few  pages*  Assume  that  we  have  the  following 
situation; 

(1)  p(e)  -  N(80,  25)        (Prior  Distribution) 

(2)  p(xle)  -  N(e,  16)       (Distribution  of  Test  Scores) 
(3) 

L(d^,  e)  I^ss  Function 


6  <  6 


e  >  e 

—  0 


(retain) 
d2  (advance) 


(4)    6   =  75 
o 


(Pass  Level) 


Before  collecting  our  observation  on  this  subject,  we  can  write  the 
posterior  probability  density  function  p(9jx)  as  a  function  of  x  .  Thus, 


P(e|x)  -  N 


80(16)  +  25x 
16  +  25 


25  +  16 
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Since  dils  is  a  completely  general  description  of  the  posterior  distribution 
no  matter  what  value  x  Is  observed^  It  should  be  possible  to  determine  those 
values  of  x  which  lead  us  to  decision  d^  and  those  which  lead  us  to  d^  * 
This  Is  what  we  did  symbolically  before  when  we  determined  the  critical  score 
X    .    We  defined  x    as  that  point  such  that  If  x  happens  to  fall  below  x  * 

0  0  o 

then  applying  our  criterion  of  minimum  expected  1osg>  we  would  be  led  to  choose 
d^  .    And  of  course^  If  x  falls  above  x^>  we  would  choose       '    "^^^  previous 
theory  tells  us  that  the  next  step  Is  to  calculate  the  expected  loss  under 
each  decision.    Doing  this  we  find: 

g      [L(d     6)]  -  /"    1  p(e|x)de 
$|x 

and 

g     a(d2»  6)]  -  sll  2  p(e|x)de  . 

By  equating  these  two  expected  values  symbolically  and  solving  for  that  value 
X  for  which  equality  obtains >  we  previously  found  that 

0900  9 
where  2    Is  the  (  )th  percentile  of  the  unit-normal  distribution. 

O  C   +  D 

Substituting  into  the  equation  for  x^,  we  find: 

^o  "  -  [25(l6)/(25  +  16)]^'}  -  80  ^ 

Therefore,  we  are  certain  that  if  we  observe  an  x  (test  score)  which 
is  greater  than  extensive  form  analysis  will  lead  us  to  advance  the 

student*    Similarly,  if  the  observed  test  score  is  less  than  extensive 
form  analysis  will  cause  us  to  retain  the  student  for  additional  training. 
Although,  obtaining  a  score  of  precisely  7A.003  will  be  somewhat  ambiguous 
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since  the  two  decisions  will  have  the  same  expected  loss,  In  practice  this 
will  not  be  a  problem*    We  will  seldom  find  a  test  score  which  falls 
precisely  on  an  Indifference  point.    Suppose  that  we  observe  ^  =  73*  Since 
73  Is  less  than       =  7A.O03,  we  can  be  certain  that  extensive  form  analysis 
will  lend  us  to  retain  the  student  (decision  d^)*    However,  for  the  sake  of 
those  who  doubt  mathematical  arguments,  we  will  calculate  the  expected  losses 
for  comparison* 

CtUd^,  6)jx  -  73]  -  /^g  1  p(6|x  -  73)d6  »  1  -  Pr(6  <  75\x  =  73) 
^^W2*  =  73]  =  fl^  Z  P(6|x  =  73)d6  =  2[Pr(6  <  75lx  =  73)]  . 

?iince 

p(6lx  =  73)  -  k(75*73,  9.756), 

we  have 

Pr(6  <  75tx  =  73)  =  .m  , 

and  so 

(^a(ap  e)ix  ^  73]  ^  *591 

and 

S[L(d2,  Q)|x  -  73]  ^  *818  * 

Clearly,  we  would  choose  d^^  and  retain  the  student  as  predicted. 

A  more  realistic  model  than  that  Just  described  would  recognize  the 
fact  that  In  educational  settings  the  model  variance  as  well  ati  the  mean 
le  usually  unknown.    In  most  situations,  the  decision  maker  will  have 
some  infontatlon  concerning  the  variation  ai\d  the  region  In  which  the 
observations  will  fall;  total  Ignorance  would  preclude  even  the  pi'oper 
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choice  of  a  measuring  Instrument*    However,  except  In  very  special  situations, 

2 

one*s  knowledge  about  the  model  variance  cr    Is  typlcaxXy  not  sufficiently 
precise  to  warrant  the  application  of  the  known  variance  model* 

The  solution  to  the  Inference  part  of  the  unknown  tnean  and  unknown 
variance  problem  has  been  provided  by  Novlck  and  Jackson  (1974)*  Beginning 
with  a  normal  model  density  with  location  parameter  9  and  variance  parameter 
^,  and  a  gatnma-normal  prior  density  proportional '^to 


b 

o 


they  are  able  to  show  that 


[m(m  +  1)]^(9  "  (mw*  +  x)/(m  +  1)] 

t(e)  -   ^  ^  r   (2*4) 

[sr  +  m(x  "  w*)^/(m  +  DP 


has  a  stuaent*s  t  distribution  on  m  degrees  of  freedom*    In  this  equation, 

R  /(m  +  1)  is  the  prior  modal  estimate  of  ^\  w*  is  the  prior  modal  estimate 

2 

of  9  given  that  <}i  =  R  /(m  +  1)  j  and,  the  parameter  m  is  a  weight  factor 
which  describes  the  decision  maker's  degree  of  confidence  in  his  estimates* 
For  the  details  of  this  development,  the  reader  is  referred  to  Novlck 
and  Jackson  (1974,  Chapter  7)* 

From  Equation  (2,4),  we  see  that  t(e)  is  linear  in  9  *  Thus, 
Pr(9  <  9^tx)  =  Pr[t(e)  <  t(9^)|xl-    And  by  using  Equation  (2*2),  we 
can  partition  the  observation  seal*  into  two  disjoint  regions;  one  which 
will  lead  to  decision  d^  and  the  other  which  will  lead  to  decision  ^2  * 
The  process  is  practically  the  ^ame  as  in  the  known  variance  case*  The 
only  Important  change  is  tl  ct:  we  now  use  the  t  table  with  m  degrees  of 
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t„  2  ,   (2.5) 


ERIC 


32 

freedom  Instead  of  a  table  of  the  unit  normal  distribution.    Thus,  we 

determine  the  point  t   which  has  percentile  rank  lOOb/(c  +  b)  and  then  ^olvn 

o 

the  equation 

tm(m  +  -  (raw.  +  x^)/(m  +  1)) 

tR^  +  m(x    -  w.)^/(m  +1)1 
o 

for       *    Although  the  process  of  solving  this  equation  will  ordinarily 
lead  to  two  reoults^  only  one  of  the  results  will  solve  Equation  (2*5)  as 
stated*    The  other  will  be  associated  with  Equation  (2.5)  with  -t^  on  the 
left-hand  sid*. 

2*4    Ail  Exaigple  vjth  a  Bioomiel  Model 

Let  ua  exaibin*  (hit  same  basic  problem  using  a  different^  and  in  some 
ways,  a  more  geniiral  obdel*    For  pedagogical  purposes,  this  analysis  will 
initially  be  of  the  ootpal  form  variety,  for  we  will  actually  exhibit  the 
Bayes  riak  function*    Latar  ve  will  redo  the  analysis  using  the  much 
simpler  extensive  form*    Sather  than  assuming  that  the  mastery  level  0  takes 
on  values  in  an  interval,  let  us  return  to  the  original  situation  where  only 
two  values  ara  possible*    As  before,  we  use  the  symbol  6^  to  denote  the  class 
of  nonmastcrfif  and  6^  the  class  of  masters  at  any  point  in  time*    On  the 
bafils  of  prior  inlormation  about  the  student  and  his  training,  the  decision 
maker  formulates  a  prior  probability  distribution  on  the  two-point  state 
space,  p(e^)  *  1  -  p  and  ^(62)  "  p>  where  0  <_  p     1  *    Thus,  p  represents 
the  **prior**  probability  that  a  given  student  is  a  master*    Clearly,  if  p 
were  equal  to  zero  or  one,  no  uncertainty  would  exist  and  no  decision  problem 
would  remain*    Ad  before^  two  actions  are  open  to  the  decision  maker;  d^, 
declare  the  atudcni  a  nom&aater  and  retain  him  at  his  present  level  or, 
declare  the  student  a  i&aater  and  advance  him  to  the  next  level.  Analogous 
to  Table  3»  our  decleion  maker  adopts  the  familiar  threshold  loss  function: 
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Table  14 
L(d^,  e^) 


(retain) 
d2  (advance) 


where  b  and  c  are  nonnegatlve* 

It  will  be  convenient  in  this  and  many  applications  to  conceptualize 
a  hypothetical  population  of  tasks  for  which  the  mastery  judgments  are 
relevant*    This  done,  we  define  a  and  6  as  the  conditional  probabilities 
of  acceptably  completing  a  randomly  selected  task  from  this  population, 
given  that  the  student  is  a  master  or  nonmaster,  respectively*    In  most 
applications  a  will  be  large  and  0  will  be  small*    Our  decision  maker 
plans  to  construct  a  mastery  test  by  selecting  t  tasks  from  our  hypothetical 
population*    For  the  purposes  of  this  example,  we  will  assume  that  the 
tasks  are  experimi^n tally  independent  and  that  the  probability  of  success 
on  each  task  depends  only  upon  the  mastery  class  to  which  the  student 
belongSs    We  denote  by  X  the  discrete  random  variable  associated  with  the 
number  of  tasks  successfully  completed*    These  assumptions  imply  that, 
given  e,  the  random  variable  X  has  a  binomial  mass  function  glv&n  by 

|T^)6''(1  -  B)^  "  for  e  - 

p(xle)  ^  \  (2.6) 
K)a''(l  -  a)^     ^  e  =  62  . 

where  x  *  0,  1,  2 ,  s  *  * ,  t  * 
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If  wo  arti  to  follow  the  plan  of  normal  fomi  analysis  previously 
outlined,  we  should  now  write  down  all  reasonable  decision  rulefi,  With 
t  -f  1  possible  outcomes  (i,e,,  x  =  0,  1,  2,  ,,,,  t) ,  an  exhaustive  list  of 
decision  rules  for  even  moderate  t,  would  be  cumbersome,  indeed.  However, 
in  the  opinion  of  these  writers  the  totality  of  reasonable  decision  rulers 
for  this  problem  can  be  summarized  by  the  relationship 


for  s  =  0,  1,  2,  t  +  1*    What  this  set  of  decision  rules  boils  down  to 

is  the  following:    Choosing  a  decision  rule  ^^(x)  is  equivalent  to  choosing 
a  cutting  score  8  on  the  teit  score  scale  such  that,  if  the  observed  number 
correct  (k)  lies  b«lov  s,  the  student  is  retained  at  the  present  level, 
otherwise  h*  is  advanced*    Selection  of  the  decision  rule  ^]^q(x),  for  example^ 
would  lead  one  to  rataln  the  student  if  he  completes;  9  items  or  less 
correctly,  and  advance  him,  if  he  obtains  a  total  score  of  at  least  10* 
For  this  problem,  the  risk  R(6  ,  6.)  associated  with  each  (6  ,  6,) 
pair  can  now  be  symbolically  represented  by 


Inserting  the  loss  function  and  model  density  into  this  relationship, 
we  find  that 


retain    if    x  <  s 


advance  if    x  >  s 


(2*7) 


t  - 


(2*8a) 


and 


t  -  X 


t  -  X 


(2* bo) 
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Thus,  the  Bayes  risk  for  each  5g  may  be  represented  by 


r(6  .  P)  =  f]  [R(5  6)] 


t  t 

=  (1  -  p)c  I  (Se'^d  -  e)''  ~   +  bpu  -  E  {b^{.\  -  " 

x=s  x=s 

t 

-  bp  +    E  [(1  -  p)c(^)e''(l  -  e)^  ^  ^  ^  bp(^)a''(l  -        ^  . 
x«s 

(2-9) 

tfow  It  Is  clear  that  If  the  expression  Inside  the  summation  sign  has  a 

structure  such  that  It  Is  positive  for  all  x  less  than  some  value  x  and 

o 

negative  for  all  x  greater  than  x^>  we  could  minimize  the  Bayes  risk, 
p)>  by  choosing  5  (x)  where  s  Is  the  smallest  Integer  such  that 
s  >^  x^  •    In  fact>  this  expression  has  the  necessary  form.    To  see  this>  all 
one  needs  to  do  Is  set  the  expression  of  Interest  equal  to  zero  and  solve 

X  t**X  X  t^x 

c(l  -  p)C)e  °(1  -  e)         °  -  bp(*')c  °(1  -  a)         °  =  0 
o  o 

for  the  value(s)  of  x    »    A  few  routine  manipulations  yield  the  root 

o 

^    ,  ^n(c/b)  -K  9x^[a  -  p)/p]  -K  t  ^n[(l  -  g)/(l  -  a)]  .  . 


Since  there  exists  only  one  root  x^>  the  expression,  thought  of  as  a  continuous 
function  of  x>  can  cross  the  x  axis  at  only  one  point.    Therefore^  It  must 
be  true  that  there  is  exactly  one  region  of  the  x  scale  where  the  expression 
Is  positive  and  exactly  one  region  where  it  Is  negative*    Although  this 
argument  assures  us  that  there  Is  exactly  one  root>  it  does  not  reveal  In 
which  region  the  expression  Is  positive  and  In  which  It  Is  negative*  We> 
therefore^  re-examine  the  expression  In  brackets  In  Equation  (2»9)» 
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For  notatlonal  convenience,  we  define  Q  to  be  the  expression  of 
Interest,  Thus, 

Q  =  (1  -  P)c66''(l  -  B)^        -  bp(Sa''(l  -  a)*^  "  . 

Also  for  convenience,  we  define  a  quantity  R  by  the  equation 

(1  -  P)cd)6''(l  -  6)^  " 

R  ;  

bpQa''(l  -  a)""  ~ 

Our  purpose  In  the  following  development  Is  to  determine  that  region  In 
which  Q  Is  negative*    As  a  first  step,  we  assume  chat  Q  <  0  and  determine  the 
implications  of  that  assumption*    If  Q  <  0,  then  by  merely  manipulating 
the  Inequality,  Q  <  0,  ve  see  that  R  <  1  and,  therefore,  iln  R  <  0  .  Thus, 


Comparing  Equation  (2.10)  and  Equation  (2.11),  we  see  that  if 
jln{[ci(l  -  6)i/[&(l  -  a)]}  >  0»  the  condition  that  Q  <  0  Is  satisfied  whenever 
X  >       *    And  If  jln{[a(l  -  &)]/[is(l  -  a)]}  <  0,  the  condition  that  Q  <  0  Is 
satisfied  whenever  x  <  x^  *    But  the  condition  that  jln{[a(l  -  6)]/[g(l  -  a)]} 
Is  satisfied  If  and  only  If  [a(l  -  &)]/[6(l  -  a)]  <  1,  which  Is  equivalent 
to  ot  <  &  »    Admitting  that  the  foregoing  Is  a  bit  confusing,  we  summarize 
the  results  In  the  following  table. 
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X  <  X  X  >  X 

o  o 


Ot  <  $ 

a  >  $ 


Q  <  0 


Q  >  0 


Q  >  0 


Q  <  0 


The  implications  of  the  statements  In  this  table  for  the  decision-making 
process  are  two* 

(1)  If  a  >       then  the  decision  maker  will  minimize  the  Bayes  risk 

^'(^g*  P)  by  choosing  6„(x)  where  s  l3  the  smallest  Integer  such 

that  s  >  X  * 
—  o 

(2)  If  a  <       the  decision  function  (2*7)  Is  Inappropriate.  The 
condition  a  <  3  implies  that  the  decision  maker's  model  asserts 
that  a  nonmaster  is  more  likely  to  get  a  particular  item  correct 
than  a  master*    Such  a  model  would  certainly  lead  the  decision 
maker  to  consider  a  decision  function  of  the  form 


advance  If  x  < 


s 


tetain  if  x  >  s 


as  more  appropriate  than  Equation  (2.7). 
Since  under  normal  conditions  ot  will  be  considerably  greater  than  3, 
Implication  number  one  above  will  usually  apply* 

Perhaps  the  most  important  characteristic  of  the  expression  for  the 
critical  or  cutting  test  score>  Equation  (2.10),  Is  Its  dependence  on  the 

probabilities  ot  and  3  *    When  ot  and  1  -  $  are  both  near  one,  the  cutting 
score  vlll  tend  to  te  small*    Crudely  speaking,  In  this  case  It  does  not 
take  many  satisfactory  performances  to  decide  whether  a  student  Is  a 
master  or  not.    On  the  other  hand  as  a  and  $  approach  one  another.  It 
becomes  ever  more  costly  Co  separate  the  masters  and  nonmastcrs.    If  s 
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is  grc^iter  than  t,  the  number  of  tasks  in  the  test  is  too  small  to  permit 
any  judgment  other  than       (retain)*    If  s  £  0,  the  only  feasible  decision 
is  to  advance  the  student*    This  might  occur  if  p,  the  prior  probability  of 
state  very  clofie  to  one*    Table  15  gives  some  examples  of  critical  or 

cutting  scores  for  some  selected  parameter  values »  with  the  loss  function 
constants  c  =  2  and  b  ■  1  in  Table  14. 

Returning  now  to  familiar  ground^  we  review  the  implications  of  the 
latest  wrinkle  in  our  decision  making  scheme  in  the  context  of  our  initial 
numerical  example*    The  only  structural  difference  between  the  present 
situation  and  that  of  Section  2*1  is  that  we  now  have  t  tasks  instead 
of  one*    Por  the  purposes  of  this  example,  let  us  assume  that  we  have  a  test 
of  length  eight  (i.e.,  t  «  8) *    Then,  following  the  procedure  outlined  for 
normal  form  analysis  problems,  we  identify  the  values  of  the  inputs  to  the 
black  box* 

(a)  Specification  of  Prior  Information*    From  Table  1,  we  see  that 
our  prior  beliefs  about  6  may  be  summarized  by  q(Q^)  =  *4  and 
pCe^)  ■  -fi-    Thus,  in  the  notation  of  this  example,  we  have 

P  =  *6,  or  the  odds  we  would  be  just  willing  to  give  that  this 
student  is  a  master  without  resort  to  current  test  score 
information  are  3/2* 

(b)  Indicating  the  Experimental  Plan*    As  we  pointed  out  in  the 
previous  development  of  this  example,  the  plan  is  for  the  decision 
maker  to  give  the  student  a  test  composed  of  8  tasks.    On  the 
basis  of  this  test,  the  decision  maker  is  to  give  the  student  a 
score  which  is  equal  to  the  number  of  tasks  correctly  answered. 

The  aasumptions  are,  of  course,  that  the  tasks  are  equally  difficult 
and  experimentally  independent,  given  e  ,    The  distribution  of  X 
given  0  can  therefore  be  described  by  the  binomial  mass  functtc»n 
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Table  15 
Cutting  Scores 
(Classify  as  a  master  1£  the  number 

correct  equals  or  exceeds  the  cutting  score) 

Prior  Prob     Number  of       Prob,  of  success  for  master  (a),  Prob  of  success  for  nonmastcr  (0) 

of  master       test  tasks        a6                a          $  a          $  a  $ 

p                   t               .8         .2              .8        .1  .9         .2  .9  .1 

.9                  10                     5                           4  6  5 

20                   10                          8        ■  12  10 

30                    15                          13  17  15 

.7                  10                     5                            5  6  5 

20                   10                          9  12  10 

30                   15                         13  18  15 

.5                 10                    6                          5  6  6 

20                   11                          9  12  11 

30                   16                         13  18  16 
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p(x|e)  ^ 


Oa^l  -  a) 


t  -  X 


t  -  X 


for  e  =  62  . 


for  e  =  e 


From  Table  2^  we  see  that  tor  this  example  a  "       and  &  ^  .2* 
This  states  that  1£  our  student  Is  a  master^  the  probability  that 
he  will  give  an  acceptable  response  to  any  task  Is  a  «  .8.  If 
he  Is  a  noniaaster^  this  probability  Is  B  "  ^2* 

(c)    Specifying  Preferences  and  Decision  Rules*    From  Table  3»  we  see 
that  in  the  notation  of  the  previous  theoretical  development,  our 
threshold  loss  may  be  described  by  c  ^  2  and  b  «  1*    In  words, 
this  Implies  that  we  would  be  twice  as  unhappy  (in  terms  of  some  measure 
of  loss)  If  we  were  to  advance  a  nonmaster  than  we  would  be  if  we 
retained  a  master — I-Cdj*  e^)/L(d^,  62)  ~  2* 

Following  the  procedure  indicated  in  our  general  development,  we  identify 
10  reasonable  decision  functions* 


for  s  -  0>  1,  2,  9. 
The  ramifications  of  these  decisions  are  summarized  In  the  following  table* 
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Decision 


Retain 


Advance 


Never 

Always 

if 

X 

=  0 

if  3t  >  1 

62  (^) 

if 

X 

_<  1 

if  X  >  2 

63(K) 

if 

X 

<  2 

if  X  >  3 

if 

X 

<  3 

if  X  >  4 

if 

X 

<  ^ 

if  X  >  5 

if 

X 

<  5 

if  X  >^  6 

if 

X 

±  6 

if  X  >  7 

if 

X 

if  X  >^  8 

59(^) 

Always 

Never 

Our  problem,  then,  will  be  to  select  from  these  10  reasonable  decisions >  that 

one  which  will  miniuiize  the  expected  or  Bayes  risk.    Substituting  into 

Equation  (2*8a)  and  (2*8b),  we  indicate  the  risk  of  each  (6^,  9j)  combination  by 

R(5^,  0^)  =  2  E  (^).2''.8^  ^ 
x=s 


and 


R(6^,         =  1  -    Z  (^).8''.2^  ^ 

X»3 


Thu3>  from  Equation  (2.9),  we  have  the  Bayes  risk  given  by 

g 

r(6^,  p)  =  .6  +    Z  [.8(^).2^.8^  "  ^  -  .6(^).8^.2^  "  ^]  .  (2.12) 
x=s 


Our  theory  tells  us  that  this  function  is  a  minimum  if  we  take  s  to  be  the 
next  integer  greater  than  x^,  where  x^  is  given  by: 


17; 
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in(2)  +  in(2/3)  +  8  iln(4) 
^0  )ln(4)  +  )ln(4) 

-=  4.104  . 

And,  therefore,  the  Bayes  risk  r(5^,  p)  will  be  a  minimum  if  we  choose  S^Cx)* 
Substituting  s  »  5  into  Equation  (2*12),  we  see  that  the  minimum  Bayes  risk 
is  r(5j,  p)  "<  *042.    To  convince  the  still  skeptical  reader  that  5^  does, 
in  fact,  lead  one  to  the  minimum  Bayes  risk,  in  Table  16  we  have  exhibited 

r(6  ,  p)  for  each  of  our  decision  rules* 

s  * 

Table  16 

r(5  .  p) 
s 

for  c/b  =  2»  p  =  .6,  and  t  =  8 
s  r(6g»  p) 


0  .800 

1  .666 

2  .397 

3  .163 

4  .051 

5  .042 

6  .123 

7  .298 

8  .499 

9  .600 
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As  promised^  we  shall  now  reconsider  the  preceding  problem  using 
extensive  form  analysis.    It  will  be  recalled  that  analyses  of  the  extensive 
form  have  two  major  advantages.    Firsts  It  Is  unnecessary  to  exhibit  all 
reasonable  decision  rules.    And  secondly^  In  minimizing  the  expected  loss» 
It  Is  necessary  to  consider  only  that  value  of  x  actually  obtained. 
As  we  shall  see»  these  simplifications  will  make  this  analysis  almost 
trivial. 

Recall  that  the  likelihood  of  X  given  9  is  given  by 


p(xiO)  = 


(^)B^(l  -  B)'^  "  ^        for  e  =  9  (noimiaster) 


(^)a''Cl  -  cc)*^  "  for  6  =  9^  (master) 


where  x  «  0»  1»  2»  ...»  t  [see  Equation  (2.6)1»  and  that  Pr(0  =        =  p  . 
Combining  these  two  probabilities  we  see  that  the  joint  probability  density 
of  9  and  X  is  given  by 


pCx»  9j)  = 


where  :c  =  0,  1,  2,  t  .    Therefore,  by  Bayes  theorem,  the  posterior 

distribution  of  9  is  given  by 


(1  -  p)b''(1  -  B)^  '     +  pa^'d  -  a)^  ' 


(1  -  p)B''(1  -  B)*^  " 


for  3  -  1 


pC9jjx)  =    <  (2.13) 

pa^'d  -  cc)^ 

for  j  =  2  . 


(1  -  p)b''(1  ~  +  pa='(l  -  a)^ 
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Combining  Table  14  with  Equation  (2,13) »  we  see  that  the  expected  posterior 


loss  under  d^^  is  given  by 


bpa'^d  -  a)*^  " 


p  Dpa  (.1  -  a; 

•^0  X      ^  (1  -  p)e  (1  -  &)  + 


and  that  under  62  by 


Q     tL(d2»  e))  - 


c(i  -  p)e''(i  -  6)*=  ' 


'elx       '  (1  -  P)e''(l  -  &f  ~    +  pa'a  ~ 


Therefore^  extensive  form  analysis  leads  one  to  the  decision  rule  given  by 


<S  (x)  = 


'd^        if  bpa^'d  -  «)*^  "     <  c(l  -  P)e''(l  ~        '  ^ 


if  bpa''(l  -  a)*^  "     >  c(l  -  p)e^(l  -  e)*^  " 


or  equivalent ly» 


" « ^"  < (f)  - (Vi  ^ ' 


^1 


->■  t  An 


1 


1 


By  the  aigument  following  Equation  (2.11) »  we  see  that  when  a  >  6  this 
decision  rul«  is  «(iuivalent  to  that  reached  under  analysis  of  the  nortwi 
form.    Thus»  when  6  >  e» 

if  X  <  X 


<S*(x) 


(■■■ 
I' 


d^         if  X  >  X 


where 
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jln(c/b)  +  Jtn[(l  -  P)/P]  +  t  jln[(l  -  3)/Cl  -  cx)] 

X  =  

^  jln[(l  -  -  a)i  +  Jtn[a/6] 

Consider  again  this  last  experimental  situation*    A  moment's  reflection 
will  confirm  that  increases  in  the  number  of  tasks  in  the  test  will  increase 
the  overall  loss  in  the  decision  problem*    Each  task  included  in  thf  test, 
for  example,  increases  the  time  the  student  must  devote  to  testing  and 
involves  some  commitment  of  facilities  and,  perhaps,  time  on  the  part  of  a 
teacher*    If  wc  can  assume  that  a  c<irtain  fixed  cost  k  is  incurred  for  each 
task  in  the  test,  we  can  reformulate  our  decision  problem  into  two  parts  to 
take  account  of  this  adaitional  loss:    The  first  being  the  selection  of  the 
critical  niimber  of  items  I  to  be  answered  correctly,  and  the  second  being 
the  selection  of  the  optimal  number  of  tasks  T  to  TUinimize  the  total  Bayes 
risk  r[6(t)  ,  p]  +  tk  [where  6(t)    denotes  the  optimum  decision  rule  when 
the  test  contains  t  items].    The  final  term,  tk,  may  be  thought  of  as  the 
cost  of  testing* 

Hithin  the  framework  of  normal  form  analysis,  we  would  seek  a  cutting 
score  (s)  and  a  number  of  test  tasks  (t)  that  would  minimize  the  total  Bayes 
risk 

s-1  t 
bp  £  (bct^d  -        "  ^  +  c(l  -  p)  Z  i^)e^a  ^  +  tk  * 

x=0  ^  x=s  ^ 

The  constant  unit  cost  or  unit  lost*  k  must  be  on  the  same  scale  as  the 
original  loss  function,  if  we  are  to  have  a  valid  total  Baye*  risk. 

For  each  fixed  value  of  t>  we  already  know  how  to  select  the  critical 
number  of  items  I  to  be  answered  correctly.    Consequently,  with  the  aid 
of  a  computer,  it  Is  easy  to  determine  values  of  r[6(t)  ,  p]  +  tk  for  a 
range  of  test  lengths  t  *    We  can  then  search  the  display  of  values  of  the 
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total  Bayes  risk,  searching  for  T»  the  optimum  level  of  t  .    An  analogous 
approach  applies  within  the  framework  of  extensive  form  analysis* 

In  our  example,  with  c  -  2,  b  -  1,  a  =  ,8,  g  =  ,2,  and  p  =  ,6,  suppose 
the  lo&s  assoclatad  with  administering  one  Item  Is  k     •01.    This  could 
happen,  for  example,  If  we  think  of  each  test  cask  as  taking  up  •Ol  as  much 
time  as  an  instructional  unit*    If  Che  loss  constant  c  ^  2  was  selected 
because  it  is  associated  with  the  time  loss  that  will  result  in  repeating 
two  instructional  units  and  b  «  1  because  it  is  associated  with  the  time 
loss  that  will  result  in  repeating  one  unlC,  then  k  will  be  on  the  same 
scale*    From  Table  17,  we  can  see  something  of  the  shape  of  the  total  Bayes 
risk  function  for  this  example*    As  the  table  clearly  IndlcateSt  in  tliia 
case  the  decision  maker  would  choose  a  test  length  of  t  =  7  and  a  critical 
test  scoPi  1=4* 
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Table  17 


Display  of  Total  Bayes  Risk 
for  Various  Values  oi  t 


No.  of  Items  (t)     Critical    r[6(t)  ,  p)  +  .01  t 


Score 
(I) 

—   -  n 
0 

1 

.6000 

1 

1 

.?900 

2 

2 

.2680 

3 

2 

.1756 

3 

.1702 

5 

3 

6 

4 

.1329 

7 

.1167 

8 

5 

.1221 

9 

5 

.1174 

10 

6 

.1248 

11 

6 

.1263 

12 

7 

.1348 

13 

7 

.1398 

U 

8 

.1489 

15 

8 

.1559 

16 

9 

.1654 

17 

9 

.1736 

18 

10 

.1833 

19 

10 

.1922 

20 

11 

.2020 

w  1;  c  "  2;  a  = 

!  .8;  e  =^  ,2 

;  p  »  .6;  k  •=  .01 
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3.    Utility  Theory 

In  this  section^  we  turn  our  attention  to  a  theory  which  is  much 
stronger  than  that  Illustrated  In  the  previous  examples*    We  will  now  abandon 
the  notion  of  loss  which  we  relied  upon  so  heavily  in  Section  2>  in  favor  of 
the  more  generally  applicable  notion  of  utility.    The  roajor  difficulty  with 
usixig  loss  functions  as  previously  described^  lies  In  the  fact  that  we  have 
slnply  assumed  their  existence*    Because  of  the  apparent  reliance  of  the 
loss  function  on  some  scale^  be  It  economic^  social^  political^  or  other^ 
It  Is  by  no  means  obvious  that  such  a  function  should  exist - 

In  marked  contrast^  utility  notions  do  not  require  that  we  invent 
a  different  and  In  some  sense  arbf^trary  scaling  procedure  for  each  problem 
we  meet.    Instead^  utility  theory  uses  the  notions  of  ordered  personal 
preferences  or  desirability  of  outcomes  to  scale  the  consequences  of  each 
(d>  9)  pair*    Although  several  axiom  systems  have  been  proposed  to  Insure 
the  existence  of  utility  functions^  these  axioms  generally  require  only 
the  very  basic  relationships  between  preferences  which  rationality  demands* 
Although  these  axioms  contain  many  structural  details  concerning  the 
nature  of  outcomes^  preferences^  and  rewards^  the  most  Important 
characteristics  of  these  axioms  for  applications  seem  to  be  the  require- 
ments for  the  comparability  of  any  two  outcomes  and  the  coherence  of  the 

3 

set  of  possible  comparisons*     The  first  requirement  merely  assures  that 
for  any  two  outcomes  A  and  B>  precisely  one  of  the  following  situations 
must  obtain; 

1)  A  Is  preferred  to  B»  or 

2)  B  Is  preferred  Wo  A»  or 


The  lnter«8tttd  «jLd  Bftthematlcally  able  reader  Is  referred  to  De  Groot 
(1970«  Chapter  7)  for  a  detailed  consideration  of  these  axioms* 
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3)    A  and  B  are  equally  desirable  outcomes  and,  ve  are,  therefore, 
indif ferent  as  to  which  occurs  - 
The  second  requirement  is  one  of  transitivity.    It  merely  asserts  that  for 
any  three  outcomes  A,  B,  and  C,  If  A  is  preferred  to  B  and  B  is  preferred  to 
C,  then  it  must  be  that  A  is  preferred  to  C.    The  point  is  that  these 
requirements  of  comparability  and  coherence  are  both  simple  and  reasonable. 
They  are  incorporated  into  our  system  of  preferences  without  question, 
for  it  is  generally  agreed  that  any  violation  of  these  axioms  in  practice, 
if  exposed,  would  be  deemed  ridiculous  and  one's  system  of  preferences 
reconsJderr ' . 

As  an  aside,  we  note  that  those  readers  who  like  the  notion  of  loss 
described  in  the  previous  section  need  not  despair.    As  Lindley  (1972)  points 
out.  In  applications  it  typically  seems  true  that  one  can  define  a  suitable 
loss  function  by 


L(d,  e)  =  Ma3c|^u(d,  e)|*  u(d,  e) 


(3.1) 


if  the  number  of  outcomes  is  finite. 

Although  we  will  not  devote  a  great  deal  of  space  to  the  problem  of 
assessing  one's  utility,  we  will  consider  one  method  which  will  work  in 
problems  \diere  there  are  a  finite  number  of  outcomes* 

We  can  represent  the  set  of  outcomes  by  the  following  table: 
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Where  C22»  say,  is  the  outcome  or  consequence  associated  with  making  decision 

when       is  the  true  state  of  nature.    Let  C    be  the  most  preferred  outcome 

and       be  the  least  preferred  in  the  table,  and  assume  that  you  are  given 

it 

a  lottery  ticket  with  a  v  percent  chance  of  "winning"  C    and  a  100  -  v 
percent  chance  of  '^winning'*  C^.    Further,  assume  that  someone  has  offered 
to  take  the  ticket  cff  your  hands  in  exchange  for  C^.^-    Your  task  is  to 
discover  that  value  of  v  such  that  you  would  be  willing  to  flip  a  fair 
Coin  to  decide  betw&en  the  alternatives^ 

1)  A  ticket  with  a  v  percent  chance  on  C    and  a  100  -  v  percent 
chance  on  C^,  or 

2)  Selling  your  ticket  for  outcome  C^^ - 

The  utility  of  outcome  C.,  can  then  be  defined  by  u(C.  J  =  v/lOO. 

ij  ij 

This  procedure  can  then  be  followed  for  each  C^^  in  turn  until  utilities 
have  been  coherently  assigned  to  each  of  the  outcomes* 

We  now  return  to  our  initial  example  to  illustrate  this  procedure. 
Recalling  that  we  have  two  states,  nonmaster  and  master,  and  two  reasonable 
decisions,  retain  asid  advance,  we  summarize  the  outcomes  in  Table  18* 

Table  18 


possible  outcomes  C^^^ 

6^  (nonmaster) 

^2  (master) 

d^  (retain) 

^11 

(advance) 

Si 

^22 

Surely  the  most  desirable  outcomes  are  C^^  and  Q*22*         either  of 
these  cases,  we  correctly  classify  the  student,  so  it  is  probably  unreasonable 
to  believe  that  one  should  be  preferred  to  the  other.    Furthermore,  on  the 
presumption  that  if  a  nonmaster  is  advanced,  he  will  not  only  lose^the 
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time  required  to  complete  the  next  unit,  but  may  also  become  frustrated 
and  discouraged,  let  us  assume  that  misclassifying  a  master  is  much  more 
desirable  than  misclassifying  a  nonmaster*    So  in  terms  of  our  notation, 
we  have  C^^  «  C^^     C    and  C^^  °       *    In  this  simplified  example,  the  only 
remaining  problem  is  to  determine  u(Cj^2)*    ^^t  we  need  to  determine  is  that 
value  V  such  that  our  decision  maker  would  be  willing  to  flip  a  fair  coin  to 
decide  which  gamble  he  will  take: 

1)  A  lottery  which  pays  off  C    (a  correctly  classified  student) 
V  percent  of  the  time  and       (a  misclassif ied  nonmaster) 
(100  *  v)  percent  of  the  time;  or 

2)  A  sure  Cj^2  (wisclassif ied  ma'^ter)* 

Admittedly,  specifying  v  is  not  an  easy  task,  but  it  can  be  done.  In 
order  to  accomplish  this,  our  decision  maker  might  be  aided  by  considering 
%ow  much  better**  or  more  desirable  correctly  classifying  a  student  is  than 
misclassifying  a  nonmaster  and  compare  this  with  how  much  better  correctly 
classifying  a  student  is  than  misclassifying  a  master*    If,  for  example, 
correctly  classifying  a  student  gives  you  10  "utiles*'  more  than  misclassifying 
a*  nonmaster  and  only  5  "utiles"  more  than  misclassifying  a  master,  then  v 
for  Cj^2  would  be  50  percent*    This  says  .hat  misclassifying  a  master  is 
half-way  between  misclassifying  a  nonmaster  and  correctly  classifying  a 
student  on  a  "utiles'*  or  desirability  scale*    Assuming  that,  in  fact,  v  =  50 
thenu(Cj^2^  "  v/lOO  =  *5*    Carrying  out  the  above  procedure  for  C^^j^,  C2j^,  and 
C22»  we  see  that  it  must  be  true  that  ^i^jj)  "  ^i^22^  "  ^        ^^Sl^  ^ 
Summarizing  this  in  Table  19,  we  have 
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Table  19 
Utilities  of  (d^,  e_j> 
(nontcaster)        82  (master) 


(retain)  1  *5 

(advance)  0  1 


In  the  next  section,  we  will  examine  some  classes  or  families  of  utility 
functions  which  may  be  used  to  describe  a  decision  maker's  preferences 
in  the  dichotomous  or  two-action  decision  problem*  ^Analogous  to  the 
minimization  of  expected  loss  in  extensive  form  analysis,  decision  theory 
with  a  utility  function  requires  us  to  select  that  decision  which  will 
maximize  the  posterior  expected  or  average  utility*    That  is,  we  Seek  that 
d^(l  =  1,  2)  such  that 

/u(d^,  9)p(0|x)de 

is     maximum*    As  usual>  9  is  tht*  parameter  which  summarizes  the  state  of 
nature  and  p(e|x)  is  its  posterior  density. 

In  what  follows^  it  will  be  Important  to  recognize  that  if 

£     [u(d.,  9)1  >    g  ^  [u(d.,  0)]  , 

elx     ^  elx 

then  for  b  >  0 

^      a      lbu(d.,  9)  +  c]  >    ^     [bu(d.,  9)  +  c]  * 
€-9|x        ^  <-9|x  ^ 

That  iflf  if  d^  ie  preferred  to  d2  using  the  utility  function  'j(<i^> 

then      will  still  be  preferred  using  any  positive  linear  transformation 

of  u(d^,  9)*    A  eimilar  demonstration  is  valid  when  the  dltoctlon  of  . 
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the  inequalities  is  reversed*    So  positive  linear  transformations  of 
utility  functions  will  not  alter  the  ultimate  decision*    This  means  that 
for  decision  purposes i  a  utility  function  needs  to  be  determined  only 
up  to  a  positive  multiplicative  constant  and  an  additive  constant* 
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4-    Utility  and  the  Two  Action  Problem 

As  we  saw  in  the  last  section,  utility  functions  are  Imprecise 
things.    In  practical  situations,  we  are  often  unable  to  specify  our 
utility  associated  with  various  9)  pairs  with  anything  approaching 

mathematical  precision.    Generally,  the  best  we  can  do  is  find  some 
approximation  which  agrees  fairly  well  with  our  subjective  evaluation 
of  the  payoffs- 

Our  purpose  in  this  section  will  be  to  Illustrate  a  variety  of 
families  or  sets  of  utility  f  unctions  which  have  proven  useful  1^ 
applications.    In  selecting  families  of  utility  functions  for  lnclu:;lon 
in  this  section,  we  have  used  two  principal  criteria.    First  of  all,  we 
have  sought  to  Include  families  which  are  mathematically  tractable  in  the 
sense  that  their  expected  values  are  easily  calculated  for  standard 
distributions.    Secondly,  we  have  sought  families;  which  permit  an  acceptable 
compromise  between  having  too  many  parameters  for  the  decision  maker  to 
conveniently  specify^  and  being  so  restricted  that  significant  aspects  of 
the  decision  maker's  preferences  cannot  be  expressed- 

A-1    Threshold  Utility 

With  threshold  utility,  like  threshold  loss  discussed  in  Section  2, 
we  f^eparate  (or  partition)  the  possible  values  of  our  state  parameter 
6  into  a  number  of  mutually  exclusive  subsets.    For  continuous  C  ve  might 
consider  the  partition  {A^^,  A^)  where  Aj^  =  {q\b  <  9^1  and  A^  =  (o|e  >_  e^} 
for  some  9^  .    Thus^  for  decision  purposest  those  values  of  9  which  are 
less  thar  "one  pointy  9^,  will  be  considered  as  a  set  and  denoted  Aj^  . 
Those  values  of  9  which  are  greater  than  or  equal  to  9^  will  be  grouped 
together  as  A2  *    This  partition  would  be  analogous  to  our  previous 
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examples  where  6  was  a  measure  of  student  ability  and  was  that  point 
which  separated  the  masters  from  the  nonmasters  on  an  educational  test. 

Another  possibility  with  threshold  utility  would  be  to  use  a  slightly 
finer  partition  of  6  •    This  might  be  accomplished^  for  examples  by  preselecting 
three  points  instead  of  one:  ^3*  ^^y*         could  then  use  the 

partition  {h^,  B^}  where       =  (eje  <  6^};       ^  ^^\^^  <^<  ^2^i 

^3      ^^1^2         ^  ^3^*         ^4  ^  ^^'^  -  ^"^^  ^  scheme  might  be  useful 

If  we  needed  to  differentiate  those  masters  who  ''just  barely  made  the 
grade"  from  those  who  had  truly  assimilated  the  material^  with  analogous 
distinctions  for  the  nonmasters*    Naturally^  even  finer  partitions  could 
be  used  if  the  situation  warranted  It* 

Returning  to  our  dichotomous  partition  ik^^  k^^}^  let  us  consld^^i  it 
in  more  detail*    If  we  denote  the  utility  associated  with  each  (d^.  A.) 
pair  by  u(d,>  A       then  in  the  two  action  problem^  we  may  represent  the 
threshold  utility  function  as  In  Table  20* 

Table  20 
u(d^,  A  ) 


Aj^(e  <  e^)     A2(6  >_  e^) 


a 


b 


c 


d 


Decision  or  action  d.  Is  then  to  be  preferred  whenever 


or  whenever 


a{pr[e  <  e  1}  +  b{Pr[e  >  e  J}  >  c{pr[e  <  e  ]}  +  d{pr[e  >  e  ]}  *  (4*i) 
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In  decision  problems  similar  to  those  discussed  earlier  in  the  co^'.text  of 
educational  testing,  the  d's  involve  decisions  about  the  true  state  of 
nature  6:    *'Is  he  a  master  or  a  nonmaster?"    When  this  is  the  case^  it 
will  usually  liappen  that  either  a  and  d  or  b  and  c  will  be  associated  with 
"correct"  decisions,  and  therefore,  ought  to  be  larger  than  the  other 
utilities  in  their  respective  colunms  In  the  table*    Since  the  labeling 
of       and       is  arbitrary,  we  will  assume  that  a  and  d  are  the  correct 
decisions  in  this  analysis.    Furthermore,  since  we  demonstrated  at  the 
end  of  the  last  section  that  rescaling  utility  by  a  positive  linear 
transformation  does  not  affect  our  decisions,  we  will  assume  that  all 
utilities  in  the  table  lie  in  the  interval  between  zero  and  one*  Applying 
these  stipulations  to  Equation  (4.1),  we  see  that  decision       will  be 
preferred  whenever 


(a  -  c){Pr[e  <  e^]}  >  (d  ^  b){Prtti  >^  e^]} 


or  alternatively. 


d  ^  b  ^      ^  0^ 


^    Pr[e  >  e  1 


Since 


we  ha  & 


d  -  b 


<  Pr[e  <  e  ]  . 


(4.3) 


X^-  c)  +  (d  -  b) 


By  a  similar  argument,  d^  will  be  preferred  whenever 


d  -  b 


>  Pr[e  <  e  1  * 


;(a  -  c)  +  (d  -  b) 
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The  reader  should  note  the  similarity  between  Equation  (4*3)  and  Equation 
(2.2)*    In  fact>  if  a  =  d     1  and  one  considers  the  loss  as  specified  in 
Equation  (3*1),  then  it  is  clear  that  the  equations  are  identical*    And  so 
by  simply  rephrasing  the  example  of  Section  2.3  in  terms  of  utility  instead 
of  loss,  we  see  that  the  posterior  cutting  score  is  given  by 


1  -  b 


2  -  b  -  c 


=  Pr[e  <  0^1x^3  . 


Consider  the  following  numerical  example  previously  discussed  in 
Section  2.3.    Suppose  that  the  test  score  X  has  a  normal  distribution  with 
unknown  mean  9  and  unknown  variance  ^  *    Further  suppose  that  after  careful 
consideration  of  all  collateral  information  available,  our  decision  maker  is 
able  to  adequately  summarize  his  prior  beliefs  about  9  and  ^  for  the  student 
under  study  in  terms  of  a  gamma-normal  distribution  with  parameters  m  9, 


w*      80,  and  R    =  144*    By  Equation  (2.4),  we  see  that 

[9(10)]^[9  -  (9*80  +  x)/10] 


t(9ix)  = 


[144  +  9(x  -  80)^/10]^ 


(4.4) 


has  a  student  t  distribution  on  9  degrees  of  freedom* 

In  order  to  apply  Equation  (4*3),  we  need  two  additional  pieces  of 
information)    A  critical  true  score  9^,  and  d  utility  function  uCd^,  9). 
As  before  we  let  9^  =  75-    For  purposes  of. this  example,  we  describe 
our  utility  function  by  the  following  table* 


u(d^,  9) 


0  <  9 

0 

e  >  9 

—  0 

dj^  (retain) 

.9 

.5 

(advance) 

0 

1 
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Applying  Equation  (4*3),  we  see  that  we  will  retain  the  student  if 

Pr[e  <  e^]  >  .5/(.9  +  *5)  »  *36*    And  by  Equation  (4.4),  this  is 

equivalent  to  ?rtt(e|x)  <  t(75|x)J  >  *36,  where  t(e|x)  is  distributed 

as  a  student^e  t  with  9  degrees  of  freedom.    If  ve  have  already  collected 

our  data  so  that  x  ia  known,  this  decision  criterion  can  be  applied 

directly*    However,  if  x  is  not  yet  known,  we  can  apply  the  same  argument 

used  in  Section  2*3  to  determine  the  now  familiar  cutting  score,  x  ,  which 

o 

divides  the  observation  scale  into  two  disjoint  decision  regions.  There 
are  two  steps  in  the  determination  of  x^  *    First^  we  must  locate  the 
point  t^  in  a  table  of  the  central  t  distribution  with  9  degrees  of 
freedom,  which  has  percentile  rank  equal  to  36.    Secondly,  we  must  solve 
the  equation 

(90)^^175  -  (720  +  X  )/lO] 
t,  -  -.37  ^  r  14.5) 


^  [lU  +  9(x    "  30)^/10)^ 


tov  X    *    Working  out  the  algebra,  we  find  x    =  43.5  and  x    =  .635. 

O  0  0 

Substituting  these  two  possible  solutions  into  Equation  (4*5),  we  see 
that  only       -  43*5  satisfies  the  equation  as  stated*    Thus,  whenever 
the  observed  x  is  greater  than  43*5,  extensive  form  analysis  will  lead 
the  deciGion  maker  to  advance  the  student*    Of  course,  whenever  x  is 
less  than  43*5,  the  student  will  be  retained* 

We  turn  now  to  another  class  of  utility  functions*    This  time,  we  will 
treat  utility  as  a  continuous  function  of  6  rather  than  a  disciatc  one. 
Continuous  utility  might  be  considered  as  the  limiting  case  of  thtesliold 
utility  as  the  partition  grows  increasingly  fine* 
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4.2    Linear  Utility 

We  direct  our  attention  first  1:0  the  simplest  of  continuous  utility 
functions^  that  is>  those  which  are  linear  in  0  .    In  the  ^.inear  two- 
action  case>  we  define  utility  ty  functions  of  the  following  form. 

e  +  f  9  i  «  1 

u(d^,  e)  =  /  (4*6) 
g  +  he  i  =  2 

The  reader  should  note  that  what  we  have  done  here  is  define  utility 

as  a  separate  linear  function  for  each  possible  decision^  d^  *    Thus,  if 

decision  one  ir  chosen^  the  payoff  or  utility  is  to  be  a  linear  function 

of  the  state  paiameter  e  with  slope  f  and  intercept  e  s    For  decision 

d^t  the  slope  is  h  and  the  intercept  is  g  s 

The  existence  of  a  breakeven  or  indifference  value  6    of  the  state 

o 

parameter  6  imposes  the  condition  that 

e  +  f9    '^g  +  he.ore    =(e-  g)/(h  -  f)  . 
o  o  o 

In  our  attempt  to  maximize  expected  utility^  we  will  select  action  d^  if 

[(e  +  fe)|xl  >  ^t(g  +  hO)|xl  . 

If       =  ligj^  denotes  the  posterior  mean  of       this  implies  that  action  d^^ 
is  taken  if 

e  +  f       >  g  +  h  * 

In  other  words,  with  linear  utility,  the  action  taken  depends  only  upon  the 
mean  of  the  posterior  distribution  of  the  state  parameter  9,  other  attributes 
of  the  distribution  are  irrelevant  for  decision  purposes* 

If  we  index  our  decisions  so  that  h  -  f  >  0,  we  take  action  d^ 
whenever       <  6    and  action  d^  whenever       >       '    ^^^^^        illustrates  this 

ErJc  193 


Figure  4.1 
Linear  Utility 

Figure  (a)  above  illustrates  linear  utility  of  the  form  of  Equation  (4.6)  with  constants  e  =  30*2,  £  =  -*4» 
g  =  -59t  and  h  =  Reparameterizing  (a)  according  to  u*(d^,  0) ^  Equation  (4*7),  the  utility  of  8^  is  equated 

to  0^  and  the  axes  are  rotated  so  that  the       branch  has  a  slope  equal  to  one*    This  reparameteri^ed  form  of  (a) 
is  illustrated  in  (b)*     In  this  illustration*  since  u*(d2,  M^)  ^  "^(dj^*  Mq)  ,  extensive  form  analysis  will  lead 
he  decision  maker  to  select  action  d^* 


situation  graphically*  When  linear  utility  is  used,  one  needs  to  calculate  only 
the  utilities  of  each  decision  at  the  posterior  mean  *  The  decision  with  the 
highest  utility  at       should  then  be  selected. 

As  the  general  linear  utility  function  now  stands.  Equation  (4*6),  we 
need  to  determine  the  four  constants  e,  f,       and  h  before  it  is  completely 
specified*    However,  if  we  employ  the  flexibility  afforded  by  the  requirement 

that  a  utility  function  needs  to  be  determined  only  up  to  a  positive  linear 
transformation,  we  can  reduce  the  number  of  unknown  constants  to  two. 
Thus,  if  h  >  0  in  Equation  (4*6),  we  may  redefine  ^i^^y  6)  by  making  the 
following  positive  linear  transformation 


where  e*  =  (e  -  g)/h  and  f  =  f/h  * 

The  nature  of  our  assumption  that  h  >  0  for  this  transformation  to  be 
valid  cannot  be  overemphasized*    The  condition  that  h  >  0  is  equivalent 
to  the  statement  that  for  decision  d^*  utility  is  a  strictly  increasing 
function  of  the  state  parameter  6  *    In  terms  of  our  previous  examples  where 
we  considered  6  to  be  an  ability  indeK,  h  >  0  would  make  sense  only  if 
were  the  decision  to  advance  the  student*    For  if       were  the  decision  to 
retain  him  at  the  present  level,  we  would  be  in  the  untenable  position  of 
asserting  that  as  ability  increases,  the  utility  or  desirability  of 
retaining  the  student  at  the  present  level  also  increases* 


u  (d^,  6)  =  [u(d^,  e)  -  g]/h  * 


And  so 


i  =  1 


(4.7) 
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If  one  is  careful  about  Toaklng  such  trai\sfonnatlons»  this  limitation 
will  not  ca*^se  serious  problems*    In  applications^  It  Is  usually  the  case 
that  the  utility  or  desirability  of  one  of  the  decisions  will  Increase 
with  the  state  parameter  6  *    Thus»  all  one  needs  to  do  Is  label  that 
decision       and  result  (4*7)  Is  completely  general* 

V?e  turn  now  to  mn  Illustration  of  one  of  the  most  direct  methods  a\'ailable 
for  determining  the  constants  e*  and  f*  of  Equation  (4*7)*    In  order  to  make 
this  method  work»  the  decision  maker  must  be  able  to  specify  two  ordered 
pairs  i&.f  9^)  and  (9^>  9i)  such  that 


Substituting  the  equivalents  of  these  expressions  from  Equation  (4*7)>  we 
have 


and 


and 


Solving  this  system  of  equations »  we  find  that 


f ♦  <■  f/h  « 


12  H 


and 


e  - 
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To  Illustrate  tht*  simplicity  of  using  linear  utility,  consider  the  following 
example*    After  completing  a  unit  of  Inctividually  Prescribed  Instruction, 
a  student  is  given  a  16  item  test  to  determine  whether  or  not  he  has  mastered 
the  material.    From  considerable  past  experience,  our  decision  maker  knows 
that  50%  of  those  students  obtaining  scores  of  12  on  the  test  are  able  to 
satisfactorily  complete  the  nerct  sequence*    On  the  basis  of  this  information, 
he  feels  that  a  "true  score**  of  twelve  is  the  minimum  necessary  for  advancing 
the  student  to  the  next  unit*    Repararaeterizing  this  true  score  in  terms  of 
proportion  correct,  we  find  that  6^  ^  *75. 

xfie  next  stage  in  our  decision  making  process  consists  of  determining 
the  posterior  distribution  on  the  state  parameter  6,  where  in  this  problem 
e  denotes  the  true  proportion  correct*    Using  the  techniques  described  in 
Sovick,  Lewis,  and  Jackson  (1973),  in  Lewis,  fexig>  and  Movick  (1973),  and  in 
Wang  (1973),  our  decision  maker  is  able  to  determine  a  posterior  distribution 
on  Y  =  sin'"'^*^  »   Although  this  distribution  is  rather  complicated  and 
apparently  docs  not  exist  in  closed  form,  its  precise  specification  is  actually 
irrelevant  for  the  decision-making  process  when  linear  utility  is  used* 
Under  linear  utility,  if  we  can  determine  or  at  least  approximate  the  expected 
value  of  e,  we  will  have  gleaned  all  the  information  from  the  posterior 
distribution  necessary  to  make  our  decision. 


Since  this  estimate  of       is,  in  fact,  equal  to  the  median  of  the  posterior 
distribution  of       it  is  likely  to  be  a  poor  estimate  only  in  those  cases 
where  the  posterior  distribution  of  6  is  highly  skewed*    Furthermore,  this 
is  likely  to  be  the  case  only  when  the  true  proportion  correct  is  near  either 


Lewis,  Wang»  and  Novick  (1973)  estimate 


according  to  the  equation 


est  of  Mg  «  sin  , 
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zero  or  one.    Thus,  as  long  as  the  critical  criterion  score  9    is  not 

o 

too  close  to  either  zero  or  one,  errors  in  estimating       are  unlikely 

to  lead  to  incorrect  advance  or  retain  decisions  of  major  consequence. 

It  is  true  that  if  the  true  proportion  correct,  9,  is  very  close  to  9^, 

«n  Incorirect  decision  is  likely.    However,  linear  utility  implies  that 

differences  in  the  utility  of  the  two  decisions  are  not  great,  for  points 

near  the  point  9^  .    What  this  means  for  the  decision  making  process  is 

that  with  a  linear  utility  function,  the  output  of  a  readily  available 

and  easy  to  use  computer  program  [see  Lewis,  Wang,  and  Novlck  (1973)]  will  enable 

our  decision  maker  to  determine  a  useful  estimate  of  the  posterior  mean 

at  which  to  evaluate  his  utility  function. 

At  the  next  stage  in  the  process,  our  decision  maker  must  specify  his 

utility  function.    Actually  in  this  example,  very  little  needs  to  be  done. 

For  most  reasonable  linear  utility  functions,  the  utility  associated  with 

the  decision  to  retain  the  student  will  have  a  smaller  slope  than  that 

associated  with  the  decision  to  advance  the  student.    Since  the  two 

branches  of  u(d>,  9)  must  intersect  at  9  ,  he  will  retain  the  student  if 
i'  o 

Ma  <  9    and  advance  him  if       ^  ^    •    Thus,  as  long  as  our  decision  maker 
Do  '  y  —  0 

is  certain  that  he  will  be  satisfied  with  a  linear  utility  function,  in  the 
dichotomous  decision  problem,  all  he  really  needs  to  determine  is  the 
ordinal  relationship  between  f  and  h  .    If  f  >  h  {i.e.,  the  slope  of 
u(dj^,  9)  is  greater  than  the  slope  of  u(d2,  9)]*  he  will  select  decision 
d-  whenever       >  6^  and  aelect  d^  wlienever       ^  ^    •         course  if 

1  U  —     O  £.  9  O 

f  <  h,  the  situation  is  reversed. 

The  ''catch'*  to  the  foregoing  simplicity  is  that  the  decision  maker  is 
usually  not  certain  that  he  will  be  satisfied  with  a  linear  utility  function 
until  he  tries  to  specify  one.    In  practice  the  utility  function  should  be 
overspecified  by  indicating  at  least  three  pairs  (9^^,  9^)  such  that 
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u(d^>  9^)  *  '^^^'j*  ^j^*         overspecl£lcatlon>  the  decision  maker  is  forced 
to  carefully  weigh  the  Implications  of  a  linear  utility  function* 

In  applications »  linear  utility  functions  seem  to  behave  more  reasonably 
in  the  neighborhood  of  the  breakeven  point,  9^,  than  do  threshold  functions* 
Xn  this  region,  the  rewards  and  penalties  for  correct  and  incorrect  decisions 
frequently  change  smoothly  rather  than  abruptly. 

The  fact  that  linear  utility  functions  are  not  bounded  when  9  is  unbounded 
creates  some  problems*    Severax  of  the  axiom  systems  that  have  been  used  to 
construct  /decision  analysis  require  that  utility  functions  be  bounded*  To 
this  theoretical  objection  must  be  added  the  practical  fact  that  unbounded 
utility  functions  simply  cannot  be  interpreted  far  from  the  breakeven  point* 
These  objections  are  partially  removed  if  the  posterior  probability  distribution 
of  the  state  parameter  9  is  fairly  closely  packed  around  the  breakeven  point* 
If  there  is  almost  no  probability  attached  to  extreme  values,  unbounded 
utility  is  of  little  practical  importance* 

4*3    Quadratic  Utility 

As  we  saw  above,  decisions  involving  linear  utility  functions  depend  only 
on  the  mean  of  the  posterior  distribution*    Quadratic  utility  functions  on 
the  other  hand,  result  in  making  decisions  that  depend  on  both  the  mean  and  the 
variance  of  the  posterior  distribution*    We  begin  by  defining  quadratic 
utility  by  a  function  of  the  form: 


^U^f  9)  = 


^-a(9  -  b)(9  -  c)  i  =  1 

(4*8) 

Vre(9  -  £)(9  -  g)  i  =  2  * 


Observe  that  in  order  to  use  this  utility  function,  we  must  specify 

the  six  constants:    a,  b,  c,  e,  f,  and  g  *    The  constants  b,  c,  f,  and  g  have 
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special  meaning  in  the  present  parameterization  of  the  utility  function* 
These  constants  correspond  to  those  values  of  6  where  the  utility  of  the 
respective  decisions  is  zero.    In  the  example  which  we  considered  extensively 
in  Section  2»  it  might  be  argued  that  one  reasonable  and  convenient  point 
which  might  be  used  to  fix  the  location  of  our  utility  function  would  be  the 
indifference  point  6^  .    If  we  can  use  the  permissible  linear  transformation 
of  u(d^>  e)  to  force  the  utility  of  6^  equal  to  2ero>  we  will  have  established 
£  reference  point  upon  which  to  judge  the  utilities  associated  with  other 
values  of  9  .    In  fact»  this  task  can  be  accomplished  by  defining 

u*(d^,  e)  -  u(d^,  e)  +  a(e^  -  b)(e^  -  c) 

or  equivalently 

u*(d^,  e)  -  u(d^,  e)  +  e(e^  -  f)(t)^  -  g)  . 
* 

Recognizing  that  at  the  indifference  point  6^^  ii(dj^>  6^)  ^  u(dr>  e^)>  we  c 
if 

rewrite  u  (d^>  e)  in  the  form 


an 


u*(d^,  e)  = 


[-a(e  -  e^)(e  -  c')     i  =  i 


-e(e  -  e^)(e  -  g')     i  =  2 


where  c'  *  b  +  c  *  G    and  g'  *  f  +  g  -  &    *    Since  our  pertaissible  linear 

0^0  ^ 

transformation  allows  us  to  specify  a  scale  for  utility  as  veil  as  a 
location^  we  My  Teduce  the  nunber  of  constants  to  be  specified  even  further 
by  a  transformation  of  the  form 

u   (di»  e)  -  u  id^,  e',/a 

for  a  >  0  .  Thus^ 

200 


ERIC 


67 


u  (d^,  e)  - 


-eM6  -  e  )(e  -  g')     1  »  2 


-(e  -  6  )(e  -  c')  1*1 


where  c*  «  b  *f  c  -  6  >  e*     — >  and  g'  ■  f     g  ^  6    *    In  order  to  determine 

o  a  o 

the  remaining  constants^  c\  e\  and  g\  the  decision  maker  needs  to  specify 
other  points  on  the  two  branches  of  the  utility  function*    Although  this 
could  be  done  directly  by  actually  specifying  the  utilities  associated  with 
(d»  6)  palrs^  we  describe  a  method  which  Is  probably  easier  to  use  In  most 
situations*    This  method  will  frequently  work  In  situations  where  the  utility 
of  each  decision  seems  to  approach  a  maximum  asymptotically  as  the  deviation 
between  6  and  6^  Increases*    In  this  case»  It  seems  reasonable  to  situate 
our  quadratic  curves  so  that  the  convex  side  Is  up  as  Illustrated  In 
Figure  4*2*    This  Is  equivalent  to  specifying  that  both  a  and  e  In  our 
original  models  Eqtiatlon  (4*8)  >  are  positive*    We  can  now  fix  two  of  the 
r^alnlng  constants  by  Identifying       and  &2  ^^^^  ^^^^  lower 
end  and  ^2  upper  end  of  the  feasible  domain  of  9  *    Since  we  have 

Indicated  that  the  utility  approaches  a  maximum  aaymptotlcally  as  6  approaches 
these  points^  6^  and  seems  reasonable  to  require  that  the  maximum 

on  the  d^  branch  occur  at  6^  and  that  the  maximum  on  the  d^  branch  occur 
at       *    This  requirement  is  equivalent  to  the  following  system  of 
equations: 


-  c')  =  0 


e 


e 


1 


A. 
de 


u   (d2»  6)  - 
e 


-€'(262  -     -  g') 


»  0 


201 


(a) 


(b) 


Figure  4*2 
Quadratic  Utility 

Figure  (a)  illustrates  a-quadratic  utility  function  of  the  form  of  Equation  (4*8)  with  constants  a  =  3, 
b  -  -1,  c  =  *85,  e  *  8.4,  f  =  .7,  and  g  =  2*0*    Reparameterizing  (a)  according  to  u    (d^,  0)  leads  to  (b)  . 
Reparameterization  u  (d^,  0)  changes  the  zero  poiat  on  the  utility  scale  sc  that  the  utility  of  0^  is  zero. 
This  is  equivalent  to  specifying  b  =  f  »  0^,  c  =  "-9^  and  g  =  1.95  in  Equation  (4*8).  Reparanteterization 
u     (d^,  0)  then  alters  the  scale  of  utility  so  that  a  =  1,  forcing  e  to  equal  2.8* 
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Thu3>  26^  -       and  g'  =  26^  -  6^  .    This  procedure  assumes,  as  is  true 

In  our  examples^  that  large  values  of  6  make  action  d^  more  desirable  and  that 
small  values  of  6  make  d^  more  favorable.    Since  the  indexing  of  the  decisions 
is  arbitrary^  however,  this  restriction  is  not  serious.    The  remaining 
constant  e*  may  now  be  determined  by  specifying  a  pair  of  state  parameters 
(9y  that  u(d^,  6^)  =  uCdj*  6^)*    Since  e'  is  the  only  unknown  in 

this  equation,  it  can  be  easily  determined. 

To  illustrate  these  computations,  let  us  reconsider  the  example  used  in 
the  previous  section  with  linear  utility.    In  this  example,  the  point  at  which 
the  decision  maker  would  be  indifferent  whether  he  advanced  (dj)  or  retained  (d^) 
the  student  was  9^  "  *75*    Since  0  is  the  "true"  proportion  correct,  the 
minimum  feasible  6  is  6^  »  0  and  the  maximum  is       "  1*0*    Thus,  solving 
c'  and  g'  in  the  equations  above,  we  find  c'  ■  ^^75  and  g'  «  1.25*    If,  in 
addition,  the  decisionmaker  feels  that  u(d^,  *7)  »  uCd^,  *85),  say,  then 
e*  can  be  found  by  solving  the  equation 

(.7  "  *75)(*7  +  .75)  -  e'(.85  -  .75)(*85  -  1.25) 

or 

e'  -  1*8  * 

The  utility  function  is  illustrated  in  Figure  ^*3.    In  general,  the  final 
decision  will  be  for  action  d^  if 


where  the  expectation  is  taken  with  respect  to  the  posterior  distribution  of 
e  *    Thus,  d^  is  to  be  preferred  whenever 


or 


**  2       **  2 
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,0         a  J  ^3  J*  *2  1*  d  1*  

  0  ^ 

Quadratic  Utility 

itit 

This  figure  illustrates  quadratic  utility  as  transformed  by  u    (d^,  9).  In 
this  example,  o*  =  -.75>  g'  =  1.25,  e'  »  1.8>  9^  ^  .75>  9^^  =  0.0,  and  62  "  i-O- 
Observe  that  u    (dj^>  9)  approaches  its  maximum  at  zero  while  u    (d2>  9) 
approaches  its  maximum  at  one*    Also  note  that  u(dj^,  .7)  =  ^id^f  -85). 
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where       and  cr^  are  the  mean  and  variance  of  the  posterior  distribution  of  the 
state  parameter  0  *    This  result  may  be  Interpreted  as  calling  for  action 
whenever  the  utility  of  d^^  at  Mg  Is  greater  than  the  utility  of  d^  at 
plus  a  correction  factor*    The  correction  factor  depends  upon  the  posterior 
variance  of  6  and  consequently  Is  a  measure  of  the  probable  deviation  of  6 
from  |ig  *    For  our  Individually  Prescribed  Instruction  exat?)les,  this  relation- 
ship has  important  consequences*    It  Indicates  that  for  decision  purposes^ 
all  we  need  from  the  posterior  distribution  of  6  Is  Its  mean  and  Its  variance* 
In  applications^  we  can  often  readily  obtain  these  values^  or  approximations 
to  them»  even  when  the  posterior  distribution  of  8  does  not  exist  In  closed 
form* 

4*4    Exponential  Utility 

Linear  and  quadratic  utility  functions  have  played  important  roles  in 
applications  of  decision  analysis*    As  we  have  seen»  linear  utility  requires 
only  that  we  evaluate  the  posterior  mean,  while  quadratic  utility  requires 
both  the  posterior  mean  and  variance*    These  simplifications  of  the  decision 
process  are  extremely  Important,  especially  when  the  posterior  distribution 
of  e  is  of  a  complicated  form*    often  we  are  able  to  estimate  the  mean  and 
sometimes  the  variance  of  6,  even  when  the  posterior  density  Itself  does 

not  exist  in  closed  form*    As  we  sh:il  see  in  this  section,  exponential 

I 

litility  also  has  a  simplifying  property  which  makes  it  particularly  useful 
with  many  of  the  standard  posterior  density  functions*    Before  illustrating 
this  special  property  of  exponential  utility,  we  will  exhibit  the  form  of 
the  function  and  perform  our  usual  simplifications*    We  define  exponential 
utility  by  a  function  of  the  following  form: 
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u(d^,  e)  - 


c  -  a  *f  exp{b9} 


c  -  f  *f  exp{-ge} 


i  ^  1 


1^2 


(4.9a) 


vhere  the  conatants  a»  b,       and  g  are  positive.    Notice  that  this  is  not 
the  most  general  form  available^  since  we  require  that  a»  b,  f»  and  g  be 
positive  and  that  the  aame  constant  c  appears  under  each  decision  rule. 
Although  this  simplification  is  made  so  that  the  estimates  of  the  constants 
are  more  eas;Lly  obtained^  it  has  certain  implications  which  the  would  be 
user  should  keep  in  mind.    This  particular  formulation  requires  that  for 
decision  d^,  utility  is  a  decreasing  function  of  the  state  parameter  0  . 
On  the  other  hand,  for  decision  d^*  utility  must  increase  witlt  increasing 
d  .    To  see  this,  all  we  need  to  do  is  rewrite  uCd^*  d)  in  the  form 
u(d2,  0)  ■  c  -  f/€xp{g9}.    Clearly  as  B  increases^  f/exp{g9}  approaches  zero. 
That  is,  as  d  increases,  the  contriburion  of  the  second  term  to  utility 
decreases  rapidly^  with  u(d2*       approaching  c  from  below  as  an  asymptote. 
A  similar  argument  with  respect  to  u(d^«  d)  shows  that  the  utility  of 
decision  d^  also  approaches  c  from  below,  but  this  time  with  decreasing  0  . 
This  relationship  is  depicted  graphically  in  Figure  A. A.    Using  our  now 
familiar  permissible  linear  transformation  to  eliminate  some  of  the  unknown 
constants,  ve  let 


a  it  ejcp{be  } 

o 

Since  a  ^  exp{b9  }     f  *f  exp{-g9       we  have 


u*(d^,  6)  - 


1 


[u(d^,  6)  -  (c  -  a  ^  exp{be^})l  . 


i  =  1 


i  -  2 


(^.9b) 


20G 


(a)  (b) 


Figure  4*4 
Exponential  Utility 

Figure  (a)  Illustrates  ^(<J^»  ^)  ftom  Equation  (4*9)  with  a  =  0*1,  b  ^  3*5,  c  =  2*0,  f  =  3*5,  and  g  =  1*24* 
Figure  (b)  Is  ^*(<J^»  9)»  the  reparameterlzed  form  of  figure  (a)*    Upon  reparameterizatlon,  the  utility  of 
6^  is  zero  and  the  scale  is  changed  so  that  c      a  =  f  =  1*0*    In  its  reparameterlzed  form,  u*(d^,  6),  aii 
exponential  utility  function  is  conpletely  determined  once       and  the  slopes  at  9^,        and  g,  of  its  two 
branches  are  specified* 
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When  the  decision  maker  turns  to  fixing  the  two  parameters  of  the  utility 

function,  b  and  g,  he  must  specify  the  precise  utility  of  at  least  one  point 

on  each  branch  of  the  utility  function  in  addition  to  S     ,    That  Is,  he  must 

o 

specify  a  pair  of  points  ((9^,  c^),  (62*  C2)),  such  that  u(d^,  9^)      c^  and 
^(^2*         ^       '    '^^^  reader  is  cautioned  that  this  is  not  equivalent  to  what 
we  have  done  in  the  past  when  we  specified  points  (6^,  9^)  such  that 
u(d^,  0^)  »  u(d2>  62).    With  exponential  utility  of  the  form  of  Equation  (A. 9), 
we  must  actually  specify  the  values  c^^  and  C2  (although  they  may,  of  course, 
be  equal)*    Once  this  is  done>  the  parameters  b  and  g  are  completely  determined^ 
for  we  have 

1  -  exp{b(9,  -  6  ))  =  c, 
1       o  1 

and 

1  -  exp{-g(62  -  9^)}  -  c^  * 
Upon  taking  logarithms  of  both  sides  and  solving,  ue  hnve 

(tn  (1  -  c^) 

"  ,  («i  -  V 

and 

Jln(l  - 

From  these  equations  and  the  restriction  on  our  model  that  b  and  g 
be  positive,  we  see  that  if  \  <  6^  and  ^2  ^  ®o'  '^l 

between  zero  and  one.    This  is  completely  reasonable,  however.    In  examining 
the  transformed  utility  function,  we  see  that  it  is  zero  at  9^  and  increases 
to  one  as  6  approaches  either  -  «>  or  +  «  for  decisions  dj  and  d^*  tospi^ctivcJv 
(see  Figure  A. 4). 

208 


(A. 10) 


(4.11) 
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Probably  the  easiest  way  to  specify       in  applications  is  to  determine 

such  that  the  utility  of  6^)  is  half  way  between  the  maximum  possible 

utility  und(ir  d    and  the  utility  of.  (d- ,  e  )*    By  specifying  a  similar  point 
J-  X  o 

62  under        the  decision  maker  can  check  his  coherence  by  observing  whether 
u(dj^,  e^^)  =  uCd^t         subjectively*    Having  established  his  utility  function, 
Che  decision  maker  will  prefer  decision  d^^  whenever 

^{1  -  exp{b(e  -  e^)}  \x]  >    £[1  -  exp{-g(e  -  e^)}  |x]  . 

But  since  expectation  is  a  linear  operator,  this  condition  is  equi^;alent  to 
preferring  whenever 

1  -  <S[exp{b(e  -  e  )}  |x]  >  1  "  (£[exp{-g(e  -  e  )}  |x] 


or 


or 


exp{"be^}  ^[exp{be}  |x]  <  exp{ge^}  ^[exp{-gd}  |x] 

exp{"(g  +  b)e^}  ^[exp{be}  |x]  <    ^[expC'-ge}  |x]  • 

But  ^[exp{t6}]  »  M(t)  has  special  significance  for  mathematical  statisticians* 
In  the  statistical  literature,  M(t)  is  referred  to  as  the  moment-generating 
function  for  6  *    And  because  of  the  importance  of  these  moment"generating 
functions,  the  integration  necessary  to  evaluate  the  expected  value  has 
been  worked  out  for  most  standard  density  functions*    Therefore,  if  the 
posterior,  distribution  of  6  is  one  of  the  standard  densities  (Normal,  Uniform, 
Triangular,  Gamma,  and  others),  the  final  stage  in  the  decisionmaking  process 
is  merely  a  matter  of  "plugging  in"  the  parameters  of  the  posterior  distribution 
on  6  and  those  of  the  utility  function*    By  reformulating  the  decision  criterion 
in  terms  of  moment*'generating  functions,  we  see  that  decision  d^  will  be 
preferred  whenever  exp{-(g  -h  b)e  }  M(b)  <  M(-g)* 

o 
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Returning  to  an  example  considered  In  Section  2*3,  we  assume  that  f:he 

2  2 

posterior  distribution  of  6  Is  normal  with  mean  Pg  =  (to    +  x0/io    +  ^)  and 
2       2  2 

variance       =  o  +  o  ) ,    Since  the  moment-generating  function  for  a  nomial 

2  2 

variable  Is  given  by  M(t)  =  exptty^  +  t  0^/2),  decision  d^  will  be  preferred 
whenever 

exptbUg  +  b^Og/2  -  (g  +  b)e^}  <  exp{-gMg  +  g^4/2) 

or 

<  9^  +  (g  -  b)o^/2  ,  (4,12) 

That  is,  the  retain  decision  is  preferred  If  the  mean  o£  the  posterior 
distribution  o£  &  Is  less  than  6^  plus  an  adjustment  which  depends  upon  the 
variance  of  the  posterior  distribution  and  the  relative  utilities  of  the 
two  decisions*    Whether  that  ad;justment  is  positive  or  negative,  depends 
upon  the  sign  of  g  -  b  *    Since  b  and  g  are  the  magnitudes  of  the  slopes  of 
the  utility  function  at  6^  for  d^  and         ^respectively,  the  difference  is 
a  measure  of  the  relative  speed  with  which  utility  is  changing  on  its  two 
branches  as  9  moves  away  from  9^  *    Thus,  If  g  >  b>  the  utility  of  d^  Is 
changing  more  rapidly  in  the  vicinity  of  fl^  than  the  utility  of  d^  ,  That 
Is,  when  g  >  b*  making  a  false,  positive  error  will  be  relatively  more 
expensive  than  an  error  of  the  false,  negative  variety  for  equal  distances 
from  9    ,    Consequently,  the  decision  maker  adjusts  his  critical  point  for 
In  a  positive  direction* 

Let  us  reconsider  a  slight  modification  of  the  known  variance  numerical 
example  presented  In  Section  2t3t    As  before,  we  assume  that: 

(1)  p(9)  -  N(80,  25)  (prior  on  6) 

(2)  p(xle)  -  N(e,  16)         (likelihood  or  model  density) 

(3)  ■  75  (critical  criterion  score). 
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Figure  4*5 
Exponential  Utility 
This  figure  illustrates  exponential  utility  in  the  form  of  Equation  (4.9b)* 
In  this  example,  we  assumed  that  0^^  =  70  and  0^  =  82*    Applying  Equations 
(4*10)  and  (4.11),  wc  concluded  that  b  is  equal  to  *14  and  that  g  is  equal 
to  .10.    TUe  posterior  distribution  of  6  is  normal  with  mean  oqual  to  83 
and  variance  equal  to  9.76. 
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We  further  assume  that  the  student  obtains  a  test  score  of  x  =  85*  Thus> 
th:i  posterior  distribution  of  t)  Is  normal  with  mean  equal  to  83  and  variance 
equal  to  9t76t 

Next>  OUT  decision  maker  mu^t  specify  his  utility  function*    On  the 
presumption  that  he  would  be  satisfied  vlth  an  exponential  description  like 
Hquatlon  (4*9)^  our  decision  maker  would  follow  the  procedures  outlined  In 
this  section*    His  first  task  would  be  to  subjectively  determine  6^  such  that 
the  utility  of  (d^,  6^)  Is  half-way  between  u(d^,  e^)  and  the  maximum  possible 
utility  on  the  d^  branch*    Next  he  must  similarly  determine  62  *  Assuming 
that  9^  "  70  and  6^  »  82  and  that  these  are  coherent^  Equations  (4*10)  and 
(4*11)  Indicate  that  b  •  *14  and  g  -  *10*    Applying  Equation  (4*12)>  we  see 
that  the  student  will  be  retained  If  ]Xq  <  [75  +  (-.08)9*76/2]  »  74*61*  Since 

"  83»  the  decision  maker  will  certainly  advance  the  student.  This  situation 
Is  Illustrated  In  Figure  4*5* 

4,5    Squared  Exponential  Utility 

The  final  family  of  utility  functions  that  we  will  consider  foi  the 
two-action  problem^  will  seem  somewhat  restricted  lii  the  amount  o£  flexibility 
that  it  permits  the  decision  maker*    However^  it  is  very  compatible  with 
normal  posterior  distributions  and  is  frequently  useful  when  other  posteriois 
may  be  approximated  by  normal  distributions*    It  is  also  a  rather  natural 
family  of  utility  functions  when  the  problem  is  an  estimation  problem  and  the 
act  and  parameter  spaces  coincide*    The  model  for  squared  exponential  utility 
is 


1  *  1  and  e  <  6^ 

0 


otherwise 


1  =  2  and  e  >  e 


0 


(4*13) 


whera  a  and  b  ara  poaitlve* 
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Turning  tc  the  specification  o£  the  parameters  a  and  b|  we  follow  the 
same  procedure  as  in  the  case  of  exponential  utility-    That  is>  we  specify 
the  utility  of  at  least  one  point  other  than  9^  for  each  branch  of  the 
utility  function.    If  these  points  and  their  utilities  are  represented  by 
the  pairs  {(e^^,  c^^) ,  (ej*  C2)},  then 


and 


1  -  cxp{-  I  (e^  -  e^)^}  -  C2  . 


Upon  taking  logarithms  and  solving^  we  find  that 


2  And  ^  c^) 


and 


2  And  - 

(82  -  e^)' 


From  these  equations  and  the  restriction  on  our  model  that  a  and  b  are  both 
positive,  we  see  that  and  C2  must  lie  in  the  interval  (0>  1).  Examining 
Figure  4*6|  we  see  that  this  is  reasonable^  for  squared-exponential  utility 
as  described  by  Equation  (4*13)  is  bounded  between  0  and  1. 

As  with  exponential  utility*  one  way  to  determine  (6^*  c^)  and  (821  C2) 
is  to  look  for  those  points  on  the  d^  and  d2  branches  such  that 

uCd^,  e^)  -  u(d^,  0^)  »  max{u(d^,  6)}  -  u(d^,  . 
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Figure  4*6 
Squared  Exponential  Utility 
This  £lgure  Illustrates  squared  exponential  utility  In  the  form  of 
Equation  (4,13)«    In  this  figure*  a  equals  «05  and  b  equals  ^01*  liote 
that  the  utility  function  ±b  bounded  between  rero  and  one^  with  u(d^t  @) 
equal  to  zero  for  6  >^  6    and  u(d«,  9)  equal  to  zero  for  ^  i  0  * 
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That  is,  try  to  determine  those  points  (d^,  6^)  which  have  desirability 
approximately  half-way  between  the  desirability  of  (d^,  e^)  and  the  maximum 
possible  desirability  under  that  decision.    It  wi'j.1  then  be  possible  to 
verify  the  consistency  (coherence)  of  these  specifications  by  comparing 
u(dj^,         ano  ^(dg*  O^)  ,  subjectively.    Tf  large  differences  are  believed 
to  exist  in  thu  payoff  of  these  two  situations,  then  some  reconciliation 
will  be  necessary. 

Action  dj^  will  bo  preferred  if  the  expected  or  average  posterior  utility 
under       is  greater  than  that  under       .    That  is,  if 


u(d^,  e)p(elx)de  > 


u(d2»  e)p(elx)dE)  * 


Since  u(d  ,  e)  and  u(d  ,  e)  are  non-zero  o!ily  in  the  regions  9  <  and 
1  ^ 

6  >  e^,  respectively,  vhis  inequality  may  be  rewritten  in  the  equivalent 


form 


f 


u(d^»  9)p(9|x)d9  >      ju(d2»  9)p(9lx) 


d9 


These  integrals  will  be  tractable  if  the  posterior  density  of  9,  p(9|x), 

2 

is  normal  with  mean       and  variance  Og  .    In  this  case, 


u(d,»  9)p(9lx) 


/2 


1  "  exp 


a(e  -  e^)- 


exp 


r  ^ 


.2^  -1 


which  may  be  rewritten  In  the  form 
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uCci^,  e)p(0|x)  «         exp  P      (e  -  ii^)^ 


exp 


/2 


(a  +  of) 


e  - 


a  +  o 


-2 


exp 


r 


ao 


a  +  a 


-2 


6 


The  first  term,  of  course,  Is  nothing  but  a  normal  density  with  mean  jig 
2 

and  variance  Gg  .    Ignoring  that  part  of  the  second  term  which  does  not  depend 

upon  6,  we  see  that  the  second  term  Is  proportional  to  a  norma]  distribution 

^2  ^2  ^1 

with  mean  (aO^  +  Og  jig)/U  +      )  and  variance  (a  +      )      .  Therefore, 

<^[u(d  ,  0)jx]  may  be  written  as 


Jo\i,of  +  a) 


exp 


ao 


,a  +  o 


e 


r 


6 


o  

Og   +  a 


7. 


exp 


e  - 


a  +  a 


-2 


d6 


But  each  of  these  integrals  is  nothing  but  the  probability  in  a  tail  area 

i  a  normal  distribution.    Therefore,  we  may  express  these  integrals  in 
terms  of  the  percentile  rank  of  6^  ,    Thus,  by  standardizing  0  in  each 
integral  so  that  each  distribution  is  unit  normal,  wu  see  that 
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ioo^lu(dj^»  e)|x) 


PR 


e  :p 


0^(0-2  +  a) 


-2 


a  +  a 


-2 


PR 


a  +  a 


-2 

6 


(a  +  ofr^ 


where  PR(z)  Is  the  percentile  rank  of  z  with  respect  to  the  unit  normal 
distribution. 

Replacing  a  by  b  and  reversing  the  limits  of  Integration*  we  have  a 
similar  expression  for  the  posterior  expected  utility  under  d^  •  Therefore* 
out  decision  criterion  may  bft  written  In  the  form:    "Choose  dj^  If 


PR(2i) 

PR(z^)  -  -  exp 


where 


flOO  -  PR(zJ) 
>  tlOO  -  PR(z^)3  -  —  exp 


-  h 


+  b 


«o-^ 


(«o  -  \y 


(A.IA) 


e  - 

o 


««o  +  "e^^'e 


a  +  a 


-2 

6 


-2 


(a  .  of 


 I2T 

(a  +  Og^)**  ° 
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e  - 

o 


^«o  +  °f 


b  +  o 


-2 


-2 


 d 


(b  +  0^') 


(b  +  o-2)*i    o  « 
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We  prefer       when  the  direction  of  the  inequality  is  reversed*  Although 
Equation  (A«1A)  looks  rather  frighteningi  it  is  really  rather  simple  to  use 
once  you  know  the  parameters  of  the  utility  function  and  of  the  posterior 
distribution  on  6  «    Of  course*  those  decision  makers  with  access  to  a 
computer  will  find  its  application  trivial* 


218 


85 


5.    A  Three  Action  Example 

Section  4  was  built  around  the  analysis  of  a  ^leclslon  problem  In  which 
two  actions  were  avallab.le  to  the  decision  maker.    Several  families  of  utility 
functions  were  studied  and  the  Ideas  were  illustrated  with  an  example  Involving 
the  decision  to  advance  or  retain  a  student  at  a  certain  level  In  a  sequential 
chain  of  instructional  steps.    Although  the  notation  becomes  more  complex  and 
the  computation  a  bit  more  tedious^  there  are  no  fundamentally  nw  Ideas  when 
we  assume  that  there  are  three  (or  any  finite  number)  of  options  open  to  the 
decision  maker.    In  this  section,  we  will  Illustrate  this  somewhat  more  general 
problem  by  using  natural  extensions  of  two  of  the  families  of  utility 
functions  discussed  earlier. 

5.1   Threshold  Utility 

Consider  the  following  slight  modification  of  the  Individually  Prescribed 
Instruction  example  discussed  In  Section  2.1.    In  the  previous  example,  when  a 
student  completed  a  unit  of  Instruction,  he  was  considered  a  master  or  a 
nonmaster  and  was  advanced  or  retained  on  the  basis  of  expected  utility. 
In  this  example,  we  merely  extend  the  number  of  levels  of  mastery  by  further 
partitioning  the  nonmasters  Into  two  groups.    The  first  group  contains  those 
nonmasters  whose  ability  Is  close  to  the  cutoff  point  separating  the  masters 
from  the  nonmasters.    The  second  group  contains  those  who  apparently  missed 
the  whole  point  of  the  lesson.    The  state  of  a  student  being  a  nonmaster  of 
the  poorer  variety  will  be  denoted  by  Q^i  the  better  nonmasters  will  be 
denoted  by  Q^*  ^nd>  the  masters  by  9^  - 

For  purposes  of  this  example,  we  assume  that  there  are  only  three  actions 
available  to  the  decision  maker.    The  student  may  repeat  b'^th  the  present  and 
the  previous  Instructional  units;  he  may  repeat  o: 'y  the  present  unit;  or,  he 
may  advance  to  the  next  unit. 
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With  these  specifications^  we  may  now  define  the  usual  utility  function  by 
the  following  table. 

Table  21 


62  63 


dj^Cback  one) 

d2(retaln) 

d^(advance) 


"11 

"12 

"13 

"21 

"22 

"23 

"31 

"32 

"33 

As  we  have  mentioned  before,  determination  o£  these  utilities  is  not  an  easy 
matter.    In  Section  3»  we  described  one  paradigm  for  their  determination 
which  might  be  helpful*    However^  we  do  not  claim  that  it  is  the  last  word 
in  utility  specification.    Nevertheless,  in  what  follows^  we  will  assume  that 
the  decision  maker  has  cohertntly  specified  the  utilitJes. 

Alter  the  test  score  x  is  available,  the  decision  will  be  made  by  selecting 
that  action  d^,  i  =  1,  2,  3  which  maximizes  the  posterior  expected  utility 

3 

I  uid  ,  e  )p(eJx)  . 
j-i        ^  ^ 

We  might  think  of  this  problem  in  terms  of  specifying  two  cutting  test  scores 
X    and  x^,  where  x    <       *    Then  for  x  <  x  ,  action  d^  will  be  taken;  for 

o  1  O  1  O  1 

X    <  X  <  Xi,  action  d^  will  be  taken;  and,  for  x  >  x- ,  action  d^  will  be  taken* 

o  1  i£  1  J 

To  determine  the  critical  points  x^  and       which  will  divide  the  range  of 

test  scores  into  a  d^^,  a  ^       '^^fii'^'^*       return  to  a  technique  described 

in  Section  2*    We  consider  the  posterior  distribution  of  9,  p(e!x),  as  a 
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function  of  X  ,    Since       is  the  indifference  point  with  respect  to  decisions 

and  at  the  expected  posterior  utility  under  must  equal  the  expected 
posterior  utility  under       .    That  is, 

3  3 


Simplifying  this>  we  see  that       should  be  determined  so  that 

^^11  "  ^3     "23  "  "21^P^^llV      ^^2     ^2  '  ^3  ^3^P<^2t^^ 


(5.1) 


Similarly^  x^^  should  be  determined  so  that 


(ujj^  -  U23  +        -  "21^P^^l'^l^      ^^32  "  "22     "33  "23^P^^2'^1^ 


+  (ujj  "  U23)  »  0  . 


(5.2) 


In  order  to  illustrate  how  to  use  Equations  (5*1)  and  (5*2)  in  applications, 
we  return  to  the  example  in  Section  2*3  where  posterior  to  our  observatioi)  x> 
the  ability  parameter  6  was  continuous  and>  in  fact>  normally  distributed* 
There  we  describe^?  the  posterior  distribution  of  6  by 


p(e|x)  -  N 


2  2 
xq    +  x<fr  ^0 

2  *  2 


For  purposes  of  this  example^  we  also  redefine  the  mastery  levels  and 
63  in  tarms  of  critical  points  T^^  and       on  the  ability  scale  (G)*    We  let 

*  (ete  <  Tj^}>  62  »  t^bj^  <  e  <  T2},  and  63  =  {e|e  >  T2}*  Then  transforming 
the  posterior  p(gIx)  into  a  posterior  on  the  normal  deviate  z»  we  see  that 
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Pr(e^jx)  =  Pr[z  <  z(T^»  x) ] , 

PrCO^Ix)  «  Pr[z  <  za^.  x)  ]  -  Pr[z  <  (T^,  x)] 

and 

PrCe^lx)  «  1  -  Pr[z  <  zd^,  x)] 

where  2 

TO    +  Xj) 
i         2  .  , 
z(T  ,  X)  =.       2  2    h  ' 

So  we  may  rewrite  Equations  (5.1)  and  (5,2)  in  the  form 


^"ll  **"  "22  "  "21  **  "12^^^^^^      ^^''^l*  ''o^^ 


+  (Uj^2  •*  "22  ~  "l3  **"  "23^^^^'^     ^(T^,  x^)] 


and 


+  ("13  -  "23^  =  ^ 


+  (^22  -  ^22  "  "^33*^  U23)Pr[z  <  ziT^,  x^)] 


•^^^33  "  "^23^  "  ^  • 


Each  of  these  equations  now  needs  to  be  solved  iteratively  for  and  • 
It  is  recommended  that  Tj^  and  T2  be  used  as  first  approximations  to  x^  and 
Xj^i  respectively* 

We  now  turn  to  a  modification  of  an  example  considered  in  Section  2»3 
to  illustrate  these  ideas*    Assume  that  we  have  the  following  situatii>n: 


89 


(1)  P(6)  -  N(80,  25) 

(2)  p(x|e)  ~  N(e,  16) 


(3)         =  60  and       =  85 


(A) 


(back  one) 

(retain) 

(advance) 


u(d^,  e^) 


Thus,  by  applying  Bayes  theorem,  the  posterior  distribution  of  6  as  a 
function  of  x  may  be  written  in  the  form: 


p(e!x)  -  N 


80(16)  +  25x  25(16) 
16+25    *  25  +  16 


Substituting  into  our  equations  for  z(T^,  x),  we  have 


60  - 


z(Tj^i  x)  = 


1280  +  25x 

41 
3.123 


9.214  -  .195x 


and 


85  - 

ziT^,  X)  =   3^^23 


1280  +  25x 


41 


a  17.218  -  .195x 


And  we  must  solve  the  equations 
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.07  PR(9.214  -  .195x^)  -  .01  PR(17.218  -  .195x^)  -1=0 
.02  PR(9.214  -  .195Xj^)  -  .07  PR(17.218  -  .195Xj^)  +4=0 
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where  PR(z)  equals  the  percentile  rank  of  z  .    Iterating  to  a  solution, 

we  find  that  x    ■  52»7  and  x,  *  87»4.    So  the  decision  maker  will  choose 
o  1 

decision  d^  and  have  the  student  repeat  two  units  if  x  <^  52;  will  choose 
decision  d^  and  have  the  student  repeat  the  current  unit  if  53  ;^  x  ^  87; 
and,  will  choose  decision  d^  and  advance  the  student  if  x  >^  88* 

5*2    Linear  Utility 

Analogous  to  the  situation  in  the  two-action  problem  (see  Section  4.2), 
we  define  linear  utility  to  be  linear  in  6  for  each  decision  separately* 
Thus,  linear  utility  in  the  three  decision  situation  is  defined  by  a 
function  of  the  form; 

fe  +  f  e  1=1 
g  +  he  i  =  2  (5*3) 

k  +  me  i  =  3 


If  we  assume  that  our  decisions  can  be  indexed  so  that  decision  d^  i$  most 
desirable  when  e  is  small*  so  that  decision  d2  is  most  desirable  when  e  takes 
intermediate  values,  and  so  that  decision  d^  is  most  desirable  when  6  is 
large;  then  the  solution  of  the  three  action  problem  is  a  straightforward 
extension  of  that  offered  in  Section  4*2*    Applying  our  permissible  positive 
linear  transformation,  for  m  >  0,  we  let  u  (d^,  9)  =  [u(d^,  e)  -  k]/m  * 
Thus, 

1  =  1 

u*(d^,  e)  =     <  g'  +  h'O         i  =  2 


where 


i  =  3 


e'  «   ;  f      ^  ;  g'  "  ^         ;  and  h*  =  — 

m  m  m  m 
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And  since  we  have  four  constants  to  estimate  In  specifying  our  linear 

utility  function^  we  need  four  ordered  pairs  (6.,  6,)  such  that 

^  2 

u  (d^>  6^)  =  u  (dy  6^)*    Two  of  these  pairs  are  provided  by  the 
breakeven  points       and  At  these  points,  we  have 

e*  +  f*T^  =  g'  +  h'T^ 

and 

gt  +  h*T2  =  T2  * 

Thus,  we  need  only  two  additional  pairs  to  completely  specify  the  utility 
function*  The  resulting  linear  system  of  four  equations  in  four  unknowns 
can  then  be  solved  for  e\  f,  g',  and  h*  * 

When  we  turn  to  maximizing  expected  utility,  we  now  have  three  equations 
to  consider*    In  fact,  depending  upon  whether  e'  +  8*  +  ^*^e»  ^6 

largest,  we  choose  decision  d^^,  d^,  or  d^,  respectively*    Graphically,  this 
is  clearly  illustrated  in  Figure  5*1*    All  the  decision  maker  needs  to  do  is 
examine  the  utility  of  each  decision  at  the  mean  of  the  posterior  distribution 
of  0,  choosing  that  decision  with  the  highest  value* 
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1^ 


7(J 


75 


Figure  5.1 

Linear  Utility  In  a  Three-Action  Problem 

This  figure  Illustrates  linear  utility  In  the  form  of  Equation  (5.3)  with 
constants  e  »  60,  f  »  -*9,  g  »  15,  h  »  -*2,  k  »  -*90>  and  m  *  1*2*    As  In  the 
two  action  problem  discussed  In  Section  4*2,  the  decision  depends  only  upon 
the  utilities  of  each  decision  at  the  posterior  mean       *    In  this 
Illustration,  since  uCd^*  Pg)  >  ^Wj*  Ug)  >  ^(dj^,  P^)*  extensive  form  analysis 
will  lead  our  declalon  maker  to  choose  action  d^*    In  terms  of       and  T,^p 
action  dj^  will  be  taken  whenever  Ug  <  T^^;  action       will  be  taken  whenever 
Tj^  <       <  T^;  and»  action  d^  will  be  taken  whenever       >  T^* 
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6.    Preposterior  Analysis 

Information  Is  never  free.    For  example^  information  about  the  mastery 
level  attained  by  a  particular  student  is  obtained  by  testing,  Interviews, 
or  class  recitation.    Such  activities  spend  the  time  of  the  student,  spend 
the  time  of  the  teacher,  and  tie  up  facilities.    If  there  are  only  meager 
rewards  and  penalties  for  correct  and  incorrect  decisions,  it  may  be  wasteful 
to  purchase  information  whose  cost  may  exceed  the  gain  in  expected  utility. 

Suppose  that  a  decision  maker  has  (1)  a  prior  distribution  on  6,  p(e), 
(2)  a  utility  function  u(d^,  e),  and  (3)  a  potential  experiment  which,  if 
carried  out,  will  have  outcomes  x  with  model  density  p(x|e).  Before 
collecting  the  data,  the  decision  maker  wants  to  know  the  e:ttent  to  which 
his  efforts  are  likely  to  be  rewarded.    That  is,  he  wants  to  know  whether 
the  additional  information  contained  in  the  potential  experiment  is  likely 
to  be  sufficiently  "valuable"  to  justify  obtaining  it.    Bayeslan  decision 
analysis  provides  the  framework  of  preposterior  analysis  for  studying  this 
question. 

Tho  logic  of  preposterior  analysis  is  simple  and  can  be  readily  under- 
stood by  considering  the  following  outline: 

(1)    The  decision  maker  can  attach  a  "value"  to  the  information 


(2)    Assume  for  the  moment  that  the  experiment  has  already  been  carried 
out  and  the  result  x  obtained.    If  this  were  the  case,  then 
analogous  to  the  above,  the  decision  maker  ctald  attach  a  value 
to  the  information  contained  in  his  posterior.    That  is. 


contained  in  his  prior,  p(6),  by  calculating  the  expected  utility 


of  the  optimal  decision.    That  is. 


Value  [p(e)]  =»  max 
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(3)  Continuing  as  i£  the  data  had  already  been  collected^  our  decision 
mnker  could  now  calculate  the  **value  added**  by  the  experimental 
results  (i-e*,  the  increase  in  expected  utility  after  the  addition 
oC  the  data) * 

Value  Added  »  max  /  u(d^,  6)p(6|x)d6    -  max   /  u(d^,  9)p(6)d6  . 

(4)  Since,  In  fact,  the  experiment  has  not  as  yet  been  executed,  o£ 
course^  the  value  added  cannot  be  determined.    Nevertheless >  the 
decision  maker  may  consider  the  value  adc'ed  to  be  a  function  of 
the  observation  random  variable  X.    In  the  jargon  of  decision 
theory^  value  added  considered  as  a  function  of  the  random  variable 
X  is  referred  to  as  the  conditional  value  of  sample  (experimental) 
information  and  is  denoted: 


v(e,  x)  =  max  /  u(d^,  6)p(e|x)de  -  max  /  u(d^,  0)p(e)d6  . 
^i  /  ^i 


It  is  conditional  because  it  can  be  calculated  only  when  x  is  known, 
(5)    Now  since  X  is  a  random  variable  with  a  probability  distribution 
p(x)=^y^(e)p(x|6)de,  it  is  clear  that  v(e,  x)  is  also  a  random 
variable.    If  the  density  of  v(e,  x)  were  a  simple  function^  it 
would  be  useful  at  this  point  to  examine  its  location  parameters 
and  even  cre<}ibility  intervals*    These  statistics  would  de3cribe 
the  decision  maker's  prior  beliefs  about  the  probable  increases 
in  utility  to  be  gained  from  sampling.    In  most  applications, 
however,  the  density  of  v(e>  x)  is  not  a  simple  function.  Although 
this  complexity  precludes  most  descriptive  indices,  in  many 
instances,  it  will  be  possible  to  determine  the  mean  of  the 
distribution  of  v(e,  x).    In  the  decision  theory  literature^  this 
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mean  is  commonly  referred  to  as  the  expected  value  of  sample 
information  and  is  denoted 

v(e)'^^^^(e,  x)p(x)dx 

whert:  p(x)  is  the  marginal  prior  density  of  X* 
(6)    The  decision  maker  may  now  compare  the  expected  value  of  sample 
information  with  the  "cost"  of  performing  the  experiment  and 
judge  whether  or  not  the  experiment  is  likely  to  be  worthwhile* 
Before  illustrating  preposterior  analysis  with  a  nuinerical  example, 
one  central  point  must  be  made*    In  step  (6),  the  decision  maker  must 
compare  an  expected  utility  with  the  cost  of  obtaining  experimental  infor- 
mation*   It  is  critical  that  these  two  quantities  not  only  be  measured  in 
the  same  units,  but  also  that  their  respective  scales  have  the  same  origin* 
If  the  expected  value  of  experimental  information,  v(e),  is  measured  in 
arbitrary  "utile"  units  while  the  cost  of  that  information  is  in  dollars 
and  cents,  a  sensible  comparison  cannot  be  expected. 

We  will  illustrate  preposterior  analysis  with  an  example.    We  let 


As  in  our  previous  examples,  75  has  been  selected  as  the  indifference  point 
between  the  acts  of  retaining  (d^)  and  advancing  (d^)  the  student.  As 
expected,  the  advance  decision  (d^)  is  positively  related  to  ability  (6) 
while  the  retain  decision  (d^)  has  a  negative  relationship* 

Suppose  further  that  the  prior  information  about  6  has  been  quantified 
in  the  form  of  a  normal  distribution  with  mean  78  and  variance  36*  Recall 
that  in  Section  4*2,  we  demonstrated  that  with  linear  utility,  the  optimum 


u(d  .  ,6)  - 
1 


-(3/5) (e  -  75)      i  -  1 


(7/5)  (6  -  75)      i  =  2 
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decii^lon  depends  only  upon  whether  or  not  the  mean  of  0  is  greater  than  0^  * 
Obviously  in  this  Cc.se,  with  only  prior  information  at  the  decision  maker's 
disposal,  the  advance  action  will  result  in  the  highest  expected  utility* 
Thus , 

Valuetp(e)]  =^^7/5)(0  -  75)p(e)de 

'  ^(^2*  ^(prior)>  =  <^^^^^^e (prior)  " 
where  ^Q(p^io^)  ^^a^  prior  on  G  *    If  the  experimenter  had  carried 

out  the  oxperimentt  the  highest  expected  utility  using  the  experimental 
results  to  help  select  the  action  would  be 


Value  [p(0|x)] 


-/(3/5)(e  -  75)p(e|x)d0  if  Pe(post)  ^  " 
J^(7/5)(e  «  75)p(elx)d0    if  P6(p^3cj  t  75 


Since  the  utility  function  is  linear  In  the  expected  utility  of  decision 
d^  is  merely  the  utility  of  the  expectation  or  mean  of  G  *  Thus, 


Value  tp(G|x)] 


The  conditional  value  of  sample  information  may  be  given  by 


v(e»  x)  = 


"^^^l*  ^(post)>  -  "^*^2»  ^e(prior)^       ^(post)  '  " 


"^*^2'  ^(post)>  -  "^^2»  ^(prior)^       ^O(post)  ^  " 


And  the  expected  value  of  sample  information  is  given  by 
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''all  K 
such  that 

^Cpost)  '  " 


all  X 
such  that 

^Cpost)  ^  " 


thus,  In  order  to  evaluate  v(e),  all  our  decision  maker  needs  to  do  is 
partition  the  range  of  x  into  two  subsets:    The  first  containing  all  x 
which  will  lead  to  a  posterior  mean  which  is  less  than  75,  and  the  second 
containing  those  x  which  force  ^Q^p^g^^j       be  greater  than  75.    If,  for  the 
purposes  of  this  example,  we  assume  that  the  likelihood  of  our  sample  may 
be  described  by  a  normal  distribution  with  mean  3  and  variance  25,  then 
applying  Bayes  theorem,  we  see  that  the  posterior  distribution  of  S  is  of 
the  form 

/nl  \     »  r36  X  +  25^78       36^25  "1 

^     L   25  +  36       '  36rr25j  ' 

And  so  the  relationship  ^g^p^g^^j  ^  ^5  is  equivalent  to  the  relationship 
X  <  72.92.    Thus,  the  expected  value  of  sample  information  is 

^  72.92 

vCe)  ^^/t^Cdi,  Me(post)>Pf^>^^      Pf^2*  ^6(post)>Pf^>^^  "  ^^^2'  ^6(prior)> 

72*92 

Let  us  pause  here  for  just  a  moment  and  examine  this  equation.    The  first 
thing  to  notice  is  that  we  are  integrating  over  a  range  of  test  scores  X 
from  -  00  Lo  +  <»  .    Conceptually,  this  may  seem  a  little  troublesome,  for  in 
most  applications,  test  scopes  are  bounded  within  a  relatively  small  range* 


98 


KGcall»  however^  that  wg  assumed  the  model  dGnsity  wa«  nornuii  in  form. 
This  assumption  implies  that  every  x  (from  -  <»  to  +       *  as  positive 
probability*    Therefore*  each  x  must  be  considered  when  taking  the  expectation. 
Although  admittedly^  tris  is  a  problem  conceptually,  in  applications  it  is 
not  very  important*    As  long  as  the  prior  distribution  on  9  is  carefully 
specified,  there  should  be  effectively  zevo  probability  that  x  will  lie 
outside  its  permissible  range* 

In  this  particular  example,  ue  have  p(e)  -  N(78,  36)  and  p(x|9)  -  N(e,  25)* 
Therefore,  p(x)  -  N(78,  61)*    So  that  in  this  case,  X  has  very  little  proba- 
bility of  falling  outside  the  range  (55,  101)*    Returning  to  our  expression 
for  the  expected  value  of  sample  information,  ue  find  two  integrals  of  the 
form; 

We  know  that  u(d^,  Ue(post)^  ^"  "e(post)*  "e(post)       ^^"^^'^  ^" 

x»  this  implies  that  "(d^^,  **e(post)^  linear  in  x  .    In  fact,  by 

substituting  Ue(post)  '  (36  x  +  25  *  78)/(36  +  25),  u(d.,  pQ^pQg,.))  be 
written  In  the  linear  form 

f-(3/5)(.59x  -  43.03)      if  i  =  1 
■ 
(7/5)(.59x  -  43.03)       i£  i  =  2  , 

and  so,  v(e)  can  be  written  as 

/^+*  72.92 
v(e)  =  /  <7/5)(.59x-A3.03)p(x)dx  -/  (3/5) ( .59x-43. 03)p (x)dx  -  u(d2»  Hg^p^.^^j) 

where  p(x)  -  N[78,  61).    It  can  "be  shown  that  the  f  ol  iciwing  relationship  helds. 
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Lex  -  d)p(x)dx  =  -  ma.)  +  ban[ii-^  ;  0,  l| 


where 


2 

p(x)  -  a  ] 


PR(2)  =  percentile  rank  of  z 
n ;  0>  ij  =  the  ordinate  or  height  of  the  unit  normal 


^  P  -  ^ 

curve  at  ^  

cr 


Using  this  relationship^  we  may  rewrite  v(e)  once  again 
v(e)  =  (7/5)  ^'^^'^\oo^^'°^^  (100  "  PR(72.92)]  +  (7/5)^/61  n[-.65:    0,  1] 


-0/5)  ^*^^^^\oo^^*°^^     PR(72.92)  -  (3/5) -/ST  n [-^65;    0,  1] 


"(7/5)(78  -  75) 

.46  . 
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1*  Introduction 

In  Individually  Prescribed  Instruction  it  has  been  proposed  (Novick, 
Lewis,  and  Jackson,  1973;  Hambleton  and  Kovick,  1973)  that  the  decision 
as  to  whether  or  not  the  j-th  student  has  successfully  mastered  a  unit  of 
instruction  should  be  based  on  the  aposteriori  probability  that  his  mastery 
proportion  (Hj)  is  greater  than  some  specified  proportion  (ir^)  and  on  the 
losses  associated  with  false-positive  and  false-negative  decisions^  It 
was  also  proposed  that  the  posterior  distribution  for  each  '^^  should 
benefit  not  only  from  the  prior  and  sample  information  on  each  person  j, 
but  also  on  the  collateral  information  gained  from  the  observations  on 
all  other  persons* 

The  rationale  for  this  klvA  of  analysis  was  first  given  in  an 
iSducational  context  by  Kelley  (1923,  1927)  and  later  reproposed  by  Novlck 
(1970),  Novick  and  Jackson  (1970),  and  by  Cronbach,  Gleser,  Nanda,  and 
Rajaratnam  (1972),    The  mathematical  structure  for  the  required  Bayesian 
Model  II  solution  was  given  by  Lindley  and  Smith  (1972), 

In  their  recent  paper,  Novick,  Lewis,  and  Jackson  (1973)  developed 
the  specific  solution  for  the  problem  of  estimating  binomial  proportions 
in  m^groups*    The  observable  random  variables— proportions  of  ''successes" 
pj  =  '^j^^j'  J  "  ^»  2»  ^»  where       and  n^  are  respectively  the  number 

of  successes  and  the  number  of  observations — were  first  mapped  into  a 
set  of  new  variables  g,  by  an  arc  sine  transformation.    The  variables 
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were  then  assumed  to  have  a  normal  density  function  with  mean 
Yj  =  sin    /ir^  and  variance       "  (An^  +  2)    ,  where  Yj  is  the  corresponding 

transformed  value  of  the  population  proportion  of  ''successes*'  tt,  *  Then 

3 

the  Bayeslan  Model  II  method  which  Is  based  on  the  exchangeability 
theorem  of  De  Flnettl  (1937)  was  applied  to  the  analysis  of  the  Indirectly 
observable  g ^ ,    In  the  Individually  Prescribed  Instruction  application, 
the  Individual  person  Is  treated  as  the  "group**  and  the  test  Items  as  the 
n  replications. 

The  validity  of  the  normality  assumption  on  the  distribution  of  the 
transformed  variables  g^  depends  on  the  sample  size  n^ .    If  n^  Is  very 
small,  the  normal  approximation  to  the  distribution  of  g^  will  not  be 
good.    In  practice,  it  was  felt  that  for      ^  8  this  assumption  will  be  very 
satisfactory  except  for  the  tails.    It  may  also  be  noted  that  the  domains 
of  the  distribution  on       and  Yj         bounded  between  0  and  —  t  while  the 
normal  distribution  has  unbounded  domain*    We  recall  that  with  a  uniform 
prior  Y^,  the  posterior  distribution  of       Is  normal  with  mean  and 
variance  (An^  +  2)  ^  under  the  above  appropriate  assumptions.    Thus,  we 
may  wish  to  check  whether  the  points  which  are  +  2  standard  deviations 
from  gj  exceed  0  and  tt/2,  respectively*    It  was  found  that  for       ^  ^ 
and  1  <^  n^  -  1,  the  points  which  are  +  2  standard  deviations  from 

the  posterior  mean  lie  within  the  (0,  ^  ^  range*    This  Implies  that  tho 
bounded  domain  of  the  distribution  of  Yj  should  not  be  a  major 
disturbance  In  considering  a  normal  approximation  to  Its  form*  We 
contend  that  In  the  m-group  procedure,  the  collateral  Information 
provided  by  other  groups  would  have  an  equivalent  effect  of  adding  more 
sample  observations  to  the  estimation  of  an  Individual  group^s  proportion. 
For  this  reason,  we  expect  that  the  violation  of  normality  In  the  cases 
of  small  sample  size  will  not  be  serious,  provided  all  nj  >_  8*  For 
smaller  sample  sizes,  a  logistic  transformation  Introduced  by  Leonard 


Q  should  be  considered,  though  this  will  require  study* 
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Regarding  the  problems  o£  varlauce  stabilization,  the  Freeman-Vuksy 
(1950)  transformation  was  considered  superior  to  Anscombe*s  (1948) 
transformation  or  the  simple  arc  sine  transformation,  especially  for 
small  sample  size  n  ,    The  condition  for  the  Freeman-Tukey  transformation 
to  stabilize  the  variance  iB  Tmi(l  -  n)  >  1  .    Namely,  the  tru*  proportion 

Ttj  should  lie  between      -  ^ — "2  ^  — ~2   '  general, 

this  analysis  should  be  very  satisfactory  provided  m  ^  15,  n  >^  8^ 
(^p  _<  .05,  and  the  above  condition  Is  satisfied. 

The  Bayeslan  procedure  begins  with  an  assutoptlon  that  the  set  of 

transformed  values  y.  Is  a  random  sample  from  a  normal  distribution  with 

3 

mean  pp  and  variance  ^p  .    The  analyst's  prior  bellefa  concerning  the 
parameters  pp,  t^p  are  partially  Incorporated  Into  the  analysis  by  specifying 
prior  distributions  for  thm.    Specifically,  pp  and       are  assumed  to  be 
Independent,  having  a  uniform  and  an  inverse  chl-^square  (with  v  d.f .  and 
parameter  X)  density  function,  respectively.    The  assumption  of  a  uniform 
distribution  for  pp  is  more  convenient  than  realistic  but  does  not 
significantly  affect  the  analysis,  provided  m  Is  reasonably  large.  We 
shall  consider  this  point  later. 

Under  the  above  distributional  assumptions  and  the  Bayeslan  specifications 
of  one*s  prior  knowledge,  the  joint  probability  density  function  (p.d.fOt 
b(g,  Y,  Mp,  tp),  of  the  Vector  variables  g'  «  (gj^,  ,  g^) , 

Y*  =  (Yj^j  Yj»  Yjjj)  snd  scalar  variables  pp  and  +p  is  obtained  as: 

b(g,  Y,  Up,  <l>p) 

^-Js(v  +  m  +  2)^^^^  j^-1^^^  ^  g^^2  ^  ^-1^^  ^    ^^^^  ^  ^^^2jj  ^^^^^ 

Novick,  Lewis,  and  Jackson  (1973)  arrived  at  an  explicit  expression  for  the 
posterior  dertisity  furtction  o£  y  given  g: 
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b(Y|g)  «  IX  +  ECy^  -  yO  V^^"*  ^    "  ^W-MivJ^Yi  -  g^)^]}  (1.2)' 

where       =  SY^/m  -    Following  Llndley's  approach,  the  joint  posterior  mode  y 
1 

was  taken  as  the  Bayeslan  modal  estimate  for  ^  *    The  components  y.  of  ^ 

were  then  used  to  provide  estimates  for  the  group  proportions  . 

The  modal  ef^tlmates  of  the  proportions  In  m  groups  taken  from  the  joint 

posterior  distribution  from  a  Bayeslan  Model  II  analysis  are  thought  to  be 

more  accurate  than  other  estimates  obtained  from  conventional  methods. 

Specifically,  the  vector  estimate  of  y  should  be  such  as  to  maximize  the 

probability  that  all  of  the  components  y    arc  near  the  tru^.  values  y . ,  i.e., 

J  j 

the  modal  estimates  niinimize  zero**one  loss  in  m  dimensionb*    In  many  applications, 
however,  one*s  primary  concern  is  to  be  able  to  reach  certain  decisions 
concerning  individual  groups  (or  persons).    This  would  be  the  case  with  a 
component  additive-squared  error  or  absolute-error  loss  function  or  component 
threshold  loss.    Rathev  than  be  satisfied  with  a  set  of  joint  estimates, 
one  would,  in  such  situations,  like  to  have  marginal  means  autl  varianccas 
and  to  make  some  probability  statements  about  each  individual's  ability 
(or  a  group's  level  of  achievement,  etc.).    In  this  context,  it  is  desirable 
to  have  knowledge  of  the  marginal  distribution  of  each  y^  .    In  the  present 
paper,  we  therefv^re  address  ourselves  to  the  problem  of  describing  the 
posterior  marginal  distributions  of  y^  .    To  maintain  certain  mathematical 
simplicity,  the  present  paper  will  deal  only  with  the  case  of  equal  n  ^ 
Even  with  this  restriction,  the  results  will  still  be  found  applicable  in 
many  educational  situations  (e.;;*,  in  assessing  students*  achievement  In  a 
course  or  Instructional  unit  by  administering  the  same  test  to  each  inember 
of  a  class) . 
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2.    Marginal  Posterior  Distributions  for  Gammas 

An  explicit  expression  for  the  marginal  posterior  density  function  for 
does  not  seem  to  be  obtainable  from  the  joint  posterior  p.d^f*,  bCvjg)* 
of  Y  •    However,  the  joint  p*d.f*  b(Y»  |t  Ppt  "frp)  given  by  (l.l),  with  v^^ 
replaced  by  v  «  (4n  +  2)  ^  for  equal  n,  can  be  integrated  with  respect  to 
each  Yj^  (1  7^  j  t  1  ^  It  m)  and       to  obtain  the  conditional  posterior 

p*d.f*,  b(Yjl'l>p,  g),  of  Yj  given  and 

b(YjUp,  g)  <^  b(Yj,  <|>p,  g) 


exp 


2v(m*j,  +  v)       \  M  +  V 


(2.1) 


where  g.  ^  ^     8j_  *    This  expression  is  readily  recognized  as  the  kernel  of 
a  normal  distribution*    Thus,  the  conditional  distribution  of  y^  given  (|>p 
and  g  is  normal  t^ith  mean 


and  variance 


v((|>-  +  m  ^v) 


Var(YjUp,  g)  --^^ 


Wow  if  <frp  can  be  considered  to  be  known  rather  precisely,  use  of  the  conditional 
distribution  will  be  justified  and  requisite  constants  can  be  obtained  from 
normal  distribution  tables.    This  will  occur  when  m,  n  are  large  (e.g., 
m  >  50,  n  >  30),  as  indicated  by  the  compulations  presented  in  Table  8 
(see  section  6).    Note  that  the  normal  integrations  with  respect  to  Up 
and  the  Y^^  will  be  valid,  provided  the  likelihood  for  these  quantities  is 
each  near  zero  outside  the  admissible  range.    With  respect  to  Up*  this  means 
that  in  the  number  of  groups  must  be,  large,  perhaps  m  >^  15.    In  the  lattt^r 
O      case,  this  means  that  the  n,  must  be  moderate,  n,  >  8. 
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Similarly,  wg  can  integrate  b(Y,       Wp,         v*r*t.  Y  and       to  obtain 
the  conditional  p.d,f*,  b(<}>j,|g),  of       given  g  (Hill,  1965;  Leonard,  ]972): 


(2.2) 


Note  that  the  second  factor  comes  from  the  prior  inverse  chi-square  distribution 

of  <jij,,  and  the  first  factor  is  derived  from  the  likelihood  of       8ivcn  its 

2 

sufficient  statistic  lig,  -  g.)  *    This  first  factor  is  the  kerne]  of  an 

i 

inverse  chi-fquare  density  displaced  by  an  amount  -v  .    A  convenient  way  to 

obtain  analytically  the  normalizing  constant,  mean»  and  variance  for  this 

distribution  of  <J)j,|g  does  not  seem  to  exist.    Hence,  direct  niimeric<il 

integration  methods  will  be  used  for  this  purpose. 

In  order  to  obtain  the  marginal  posterior  p*d.f.  for  y  »  one  woultl 

J 

multiply  the  conditional  p,d.f*  of       given  <J>j,  and  g  and  that  of  given 
as  formulated  in  (2,1)  and  (2.2),  and  integrate  the  result  w.r.t.        •  Again, 
an  analytical  solution  to  this  problem  docs  not  appear  to  be  possible.    It  is 
necessary,  therefore,  to  resort  to  numerical  integration  methods  Eor  computing 
the  marginal  posterior  means  and  variances  of  Y^lg  *    For  this  task^  the  simple 
form  of  b(Yji^j,,  g)  is  helpful  in  reducing  the  required  computatioufil  efforts. 

The  computational  procedure  we  propose  begins  with  the  fiu  t  that  llic  r-lh 
moment  of  Yjig  equals  the  expected  value  (taken  over       8ivcn  g)  of  the 
conditional  r-th  raw  moment  of  y    given       and  g,  viz. 


(2.3) 


In  terms  of  (2.3),  the  marginal  posterior  mean  of  is  computed  by  the 
following  equation: 
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§  )  -  p  g  +  (1  -  p  )g. , 

(2. A) 


where  p 


We  note  that  0  <       <  1,  hence,  (2«A)  Is  In  fact  a  weighted  average  of  the 
values  gj  and  g*  *    For  notatlonal  convenience,  we  shall  write  (frp  for  <|>p|§  In 
the  sequel*    Likewise,  the  expression  V^.  f(<|>p)  Is  understood  to  be  the 
conditional  expectation  of  the  function  f(<l>p)  given  g  . 

The  posterior  variance  of       l^s  obtained  from  the  relation: 


'  Var(Yjlg)  -         lVar(YjUp,  g)]  +  V::r[£  (Yj  Up,  g)] 

r  <frp 

Thus,  computationally,  we  use: 


(2.5) 


A  A  '^1 

p    V  +  (1  -  p  )m   V    for  all  j  "  1,  m; 


(2*6) 


and 


Vart^(yjUp,  g)] 
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(2.7) 


To  study  the  characteristics  of  the  marginal  posterior  distribution  of 
Yj>  one  would  also  like  to  compute  its  coefficient  of  skeviness.    For  tltis,  we 
first  find  the  third  central  moment  QCVjlg)  of       given  g  from  the  general 


formula: 


+  3  CovtVar(Y.Up,  g),  <2,(yJ*p>  8)1  ,  (2.8) 

where  Gov  denotes  a  covarlance*    In  the  present  case,  Q(Yjl<!*pt  g)  =  0 
since  the  conditional  posterior  distribution  of       given  <}>p  and  g  is  normal 
[see  equation  (2*1)]*  Furthermore, 


*pg.  +  vg. 

 J  u 


-l3 


3 
Z 
A»0 


C2.9) 


where       [^(Yjl^p.  g)  1  is  the  third  central  moment  (w.r.t.  ^j,)  of  tha 


conditional  expectation  ^(YjI^pj  g) ,  and 

J    1  - 
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Cov[Var(YjUj.»  g),  ^  (Y.Up»  g)  1 


v(*p  +  v/m)    ^  ^pg^  +  vg. 


*p  +  V 


4r  +  ^ 


-12  2 
ni    V  (gj-M^)+v  (g'-Mj) 


r 


(2,10) 


Hence,  in  terms  of  equations  (2,5)  to  (2,10)  >  one  finds  for  the  coefficient 
of  skevness  6^  of  the  marginal  posterior  distribution  of  Yj  given  g: 

.3/2 


S  "  Q(Yjlg)/[Var(Y^lg)]^ 


In  sunsnaiy^  it  is  seen  from  equations  (2,4)  to  (2,10)  that  given  the 

it 

expectations  with  respect  to       of  the  functions  ^p(4^j*  v) 
(0  £  it  £  k>  k  =  1>  2>  3)  of       and  the  Indirectly  observable  vector  g> 


the 


descriptive  statistics  of  our  interest — the  mean  ]i variance         and  index 


can 


of  skewness  6^ — for  the  marginal  posterior  distributions  of  the  Yj 
be  easily  computed.    To  obtain  the  values  £^^[*J(*p  +  v)"*^),  we  use  numerical 
Integration  methods.    First,  the  ri^t-hand  side  (r,h*s,)  of  (2,2)  is 
integrated  w,r,t,  4>p  (0  <  (|>p  <  <*>)  ^  and  the  reciprocal  oi  the  resulting  value 
is  taken  to  give  the  proportionality  constant  for  b(4plg)  in  (2,2),  The 
particular  integration  algorithm  adopted  here  is  one  which  applies  Simpson* 8 
rule  and  uses  local  parabolic  fitting  to  the  curve  being  integrated  in 
computing  the  partitioned  integral  over  a  small  range  of  the  argument  (in 
this  case>  ^^)*    For  detailed  information^  the  reader  may  refer  to  Ralstf 
(1965,  p,  119), 
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The  next  step  involves  computing  the  expectations  of  f .  = 
^r^^r  *    ^'^^  same  integration  algorithm  described  above  is  employed 

to  obtain  t-^^^^  j^^^r^  ^  ^o^JL^k^^P^'^^^r'^^^^r  '  variance,  and 

index  of  skewness  for  the  marginal  posterior  distribution  of       ^tre  then 
obtained  via  equations  (2*^)  to  (2.10).    A  Fortran  TV  program,  MARPRO,  was 
written  to  carry  out  all  these  computations. 

Finally,  the  exact  posterior  probability  probC^r.  >^  u  jg)  that  the  j-th 
group's  proportion  is  greater  than  or  equal  to  some  prespecified  tutting 
point  ir^  given  the  observed  vector  g  can  also  be  calculated.  Explicitly, 


prob(7T^  I^qU) 
=  prob(Y^  >  Y^lg) 


=  r  b(Yjg)dY 
^o 

/^b(YjUp,  g)b(4»plg)d4»j,dYj 

where  y    "  sin  ^(*^  )  is  the  arc  sine  transformation  of  tt    ,    The  inner 
'o  o  o 

integral  for  given       is  recalled  to  be  the  upper  end  cumulative  normal 
probability  since  b(Yj|<tpt  g)  is  a  normal  density*    The  outer  integral 
(v.r*t*  (J>p)  is  obtained  using  the  same  numerical  integration  algorithm 
described  earlier  in  this  section.    The  program  MARPRO  also  provides 
these  probabilities  with  various  values  of       (for  ,95  ^      ^  *05  in' 
steps  of  *05,  terminating  with  a  value       for  which  probCiTj  >^  ir^)  >^  ,99)* 
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3.    Marginal  Mean  Estimates  as  Compared  to  Joint  Modal  Estimates 


In  the  Novlck,  Lewis,  and  Jackson  (1973)  paper,  the  joint  posterior 
modal  estimate  (Yj)  for       (the  arc  sine  transformation  of  the  proportion 
Hj)  Is  obtained  as  a  weighted  average  of  g^  (the  corresponding  transformation 
of  the  observed  proportion  p  )  and  the  average       of  the  estimated  values 
In  m-groups.  Explicitly, 


where 


Pjgj  +  (1  "  Pj)Y.  . 


X  +  S(y,  -  yJ 
1  ^ 


m  +  V  -  1 


1 


m  +  V  -  1 


+  V. 


and  y. 


(3.1) 


-1 
m  Sy. 


In  the  case  where  all  m  groups  have  same  sample  sizes,  n^  =  n,  equation  (3.1) 
can  be  simplified  as 


=  Pg^  +  (1  -  P)y»     Pg.  +  (1  -  p)g.  , 


(3.2) 


since  now  Y*  "  g*  •    Here  p  can  be  obtained  as  the  solution  of  a  cubic  equation 
[Novlck,  Lewis,  and  Jackson,  1973,  p.  37,  (6.18)].    It  may  be  recalled  that  a 
parallel  expression  for  the  marginal  posterior  mean  (m.)  o£  y  [equation  (2*A)1 
was  obtained  In  the  previous  section.    There  the  weight  p*  Is  the  conditional 
mean  (w.r*t.  <fry)  of  ^ — given  g  .    (All  estimates  concerning  us  hereafter 
are  understood  to  be  the  posterior  estimates  so  that  the  word  ^V^sterlor**  will 
be  omitted  In  the  sequel.) 

Returning  to  (3.2),  we  may  write 
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where  <f-  *       +  ^i^j  ^  7*)  J/(m  +  v  -  1)  is  an  estimate  of       .  Thus, 
both  equations  (3*2)  and  (2*4)  are  special  forms  of  the  Kelley  type 
formula  (Kelley,  1927)*    The  only  difference  is  that  p  is  an  estimate 
of  the  reliability  R(  =  ^p^^r  ^  based  on  an  estimate  of  the 

variance        while  p    Is  the  expected  value  (over        of  R  given  g 
(i*e*,  a  Bayesian  mean  estimate  of  R  w*r*t*  <}>p)  * 

At  this  point,  we  are  Interested  in  comparing  the  ntarglnal  mean  estimates 
^       of       (or  equivalently       of  the  proportions  t/)  with  their  joint 
modal  estimates  (Yjj  or,  equivalently,  i^)*    This  comparison  relics  solely  on 
the  relative  magnitudes  of  p  and  p*  *    We  have  found    rom  our  numeric<il 
investigation  that  p    is  substantially  larger  than  p  for  moderate  n  . 
This  means  that  the  marginal  mean  estimates  are  less  rt^gressed  tow<irds  the 
common  value  g*  than  the  joint  modal  estimates^    Similarly,  we  vould  expect 
that  the  marginal  modal  estimates  would  be  less  regressed  to  tho  common 
than  the  joint  modal  estimates*    In  particular,  the  marginal  modal  estimates 
coincide  with  the  marginal  mean  estimates  when  the  marginal  distributions 
are  unimodal  and  symmetric*    In  the  present  context,  the  nari^inal  dit^Lribution 
of  Yj  given  g  is  unimodal  and  nearly  symmetric*    More  discussions;  on  Lhe 
shape  of  these  distributions  will  be  given  in  section  5,    To  elahnrate  the 
above  results,  let  us  rewrite  equations  (2*4)  and  (3*2)  as: 

r  (Yj|g)  ^  Yj  ^  g.  +  p*(gj  -  g*)  , 

and 

=  8*  +  PCgj  -  gO  . 

Then  it  is  obvious  that  if  for  a  particular  gro\ip  j  its  observed  gventi^i 
than  g*,  we  have  Y*  ^  Vj>  conversely  for       5  g*  we  tlnd       i        *     In  ivmu 
of  proportions,  we  obtain 
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and 


TTj  <  Tl^  if        <  g*   ,  (3*3) 


where       and       are  estimates  of  the  proportion       based  on       and  yy 
respectively*    Numerical  illustrations  of  the  relation  (3.3)  are  given  in 
section  6. 

The  reader  is  again  reminded  that  the  problem  of  estimation  is  closely 
liTiked  with  the  concept  of  loss  function*    Different  estimators  are  chosen  for 

different  loss  functions*    The  substantial  discrepancies  found  between  the 
joint  modal  estimates  and  the  marginal  mean  estimates  of       suggest  that  the 
defined  loss  function  and  the  kind  of  decision  (an  overall  dtcisioa  for  all 
groups  or  decisions  to  be  made  on  individual  groups  separately)  are  important 
in  the  present  estimation  problem*    If  one  is  primarily  interested  in  making 
an  overall  decision  for  all  groups  (persons,  in  many  applications  to  the 
educational  assessment  practices)  and  zero^one  loss  is  choseni  he  would  take 
the  joint  modal  estimates.    On  the  other  hand,  if  individual  decisions 
are  the  main  concern  ^d  squared-etror  loss  is  considered  appropriate,  he 
would  choose  to  use  the  marginal  mean  eatimatei*    For  individual  decisions 
with  zero^one  loss  the  marginal  nodes  would  be  the  ideal  estimators.  Hbweveri 
in  the  present  context,  these  marginal  modes  would  likely  be  close  to  the 
marginal  means* 

One  final  comment  on  the  effect  of  sample  size  n  .    The  reliability 
R  =  <{>p(<{>p  +  v)"^  increases  as  n  becomes  larger,  since  v  -  (An  +  2)  ^  decreases. 
Hence,  both  p  and  P*  (being  estimators  of  R)  are  also  expected  to  increase 
with  n  *    In  the  limit  (n     «>)  ,  both  will  approach  unity.    That  is,  our 
estimates  will  be  based  completely  on  the  observed  values  gj  -    For  this 
sama  reason,  the  estimate       and       (or,       and  n^)  will  differ  less  for 
larger  n  *    On  the  other  hand,  as  m  increases,  more  collateral  information 
is  available.    One  would  then  be  likely  to  shift  more  weight  to  the  common 
value  in  obtaining  estimates  for       .    Detailed  numerical  examples  are 
provided  in  section  6*  ^OX 
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A,    Some  Limiting  Distributions  for  Gamma 

In  the  Bayeslan  estimation  of  ra*-group  proportions,  it  is  said  that  the 

remaining  m  -  1  groups  in  effect  provide  some  sort  of  "prior  information** 

(strictly  speaking,  collateral  information  since  it  is  not  obtained  prior 

to  analysis)  for  estimating  the  proportions  in  an  individual  group.    In  view 

of  this  statement  one  may  hope  to  find  an  approximate  expression  for  b(Y.lg) 

1  ^ 

by  first  working  with  the  posterior  p,d,f,  bCVjIv*,  g)  of       given  assuming 
the  vector  y  '  =  (y^^,  y^^  ,,,,       _       '^'j  +  1'  '^m^       known.    This  posterior 

distribution  can  be  obtained  from  equation  (1.2)  by  making  the  substitutions: 

Y.  -  Y.  =^  Cy,  -  Y^) 

and 

i    i  i  m  3 

where 

Y*  =  (m  -  l)'-^    I    Y.  . 
Thus,  we  arrive  at 

b(Y Jy,  §)  «  [X  +    2  (y,  -  yO    +  — - —  (y.  -  yO  ] 

*  {exp[-i5v^"-^(Yj  -  ^P^V  (4.1) 


since 


bCYjl/,  g)  «  b(Ylg)  . 


The  second  factor  of  the  r,h,s*  of  (A.l)  is  the  likelihood  of  y^  given 
gj  and  the  first  factor  can  be  regarded  as  the  contribution  from  the  i^rior 
information  about  y^  provided  by  y    (in  addition  to  X  and  v,  of  course). 
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Therefore^  the  posterior  distribution  of       given  §  is  explicitly  available 
if  ^*  is  indeed  known.    In  reality^  it  is  unlikely  to  know  7*  beforehand. 
At  first  thought,  one  may  be  tempted  to  substitute  the  Joint  modal  estimates 

(which  are  relatively  easy  to  obtain  as  given  in  Novick>  Lewis>  and  Jackson, 
1973)  for  Yj_  (i  ^  J)  in  (4.1)  to  find  an  approximation  for  b(Yj||).  This 
expedient  8tep  is  appealing  because  only  the  mean  y^  ^nd  sum  of  squared 
deviations    Z  (y    -  y.)    enter  to  the  density  function  (4.1).    This  approach 
was  tried  but  found  to  be  insufficiently  precise. 

Returning  to  equation  (4.1),  it  is  noted  that  the  first  term  of  its 

r*h*s.  is  the  kernel  of  a  nonstandardized  t-distribution  with  d.f.  v*  *  v  +  m  -  2 

*  ~1  *  2 

and  parameters  ?  *  Y-*  <  =  ^dn  -  1)     [X  +    D  (y.  ^  Y-)  1-    (See  Novick 

and  Jackson,  1974*)    When  m     <»,  this  t-distribution  approaches  a  normal 

distribution  with  mean  y^  and  variance  <}>    =  {X  +    Z  (y,  ~  y*)^J/(v  +  m  -  4)  . 

Consequently,  b(YjiY>  g>  ra     <») ,  being  proportional  to  the  product  of  a 

normal  likelihood  and  a  normal  prior  density,  is  itself  a  normal  density. 

We  conclude  from  this  standard  Bayesian  result  that  the  limiting  (m  <») 

posterior  distribution  of  Yj  given  g,  for  known  Y  >  is  a  normal  distribution 

with  mean      (YjIy**  g>  m <»)  »  (<{>*gj  +  v^y*)/(4>*  +  v^)  and  variance 

Var(YjlY*>  §,  m     «*)  "  Vj(^*/((^*  +  v^).    Unfortunately,  this  limiting 

distribution  is  not  very  useful  in  practice  since  Y    is  not  typicallly 

known* 

A  second  related  limiting  distribution  which  might  be  of  interest  is 
that  of  Yj  given  g  when  both  w  and  n  tend  to  infinity*    For  equal  sample  size  n, 
integrating  the  Joint  p*d*f.  b(Y,  g,  pp,         [equation  (l.l)]  w.r.t.  each  y^ 
(i  J*  J)  and  then  w*r.t*  pp,  we  obtain  the  joint  posterior  p. d.f*  of  Yj  ^ind 
<}>P  given  g: 
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where 

_     -  2 

f,(<tp)^-=  Wr  +v)        ^       exp{-!i(<t„  +v)"^    I  (g,  -  g*)^}  , 

and 

(Note:    g*      (m  -  1)      Z  g.  and  b(<J>p)  is  the  prior  p.d.f,  of 

As  n  the  contribution  of         0)  and  in""'*v(->'  0)  to  fj^  and 

in  (4,2),  relative  to  that  of  <J>j,  will  become  negligible.    It  follows  that 
^l^^r*  '^j^^2^^r^^3^^r^  approicimated  by 

^  -*£(v  +  m  +  1)      /  1  A  "Ir^  _L  r  *\2  J.  m  -  1  .  ^*\2.^ 

'exp{'h^„    [X  +  Z  (g    -  gO    +  — ;; —  (y    -  g.)  ]}  , 

so  that 

MX  +    Z    (g.  -  g*)2  +  "LZJ.  (      „  g*)2r^^^  - 

as  n  +     ,    Thus,  b(Yj|g)  =  *l'plg)'*^r       approximately  proportional 

to 

expt-*sv    (y   -  g  )  ]  [X+    I  (g^  -  gO    + — (y.  -  g.)  1  » 

(4.3) 

when  n  -»■  »  . 

If  we  further  let  m     »,  the  second  factor  of  the  expression  (4.3), 
being  the  kernel  of  an  unstandardized  t  density,  approaches  a  normal  density 
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with  mean  g.  and  variance  <J>      {A  +    D  (g    -  g.)  }/(v  +  m  -  4).  The 
expression  (4.3)  Is  parallel  to  the  r.h-s.  of  (4.1)  with  y^(±  ^  j)  and 
Y.    replaced  by       (1  r  j)  and  g.  .    On  the  same  ground  discussed  In 
connection  with  hiy^\y  ,  g,  m  -i-  •) ,  It  Is  then  obvious  that,  when  m,  n  -i-  «, 
the  posterior  marginal  distribution  of       given  g  Is  normal  with 


mean 


^(Yj  Ig,  m,  n     «)  »  ($gj  +  vg?)($  +  v)""*"  and  variance 


1A  ^  ^1 

g,  m,  n     «)  ■  <J>v($  +  v)      ,    This  simple  form  of  the  limiting 

distribution  for  b(Yjjg)  suggests  exploring  the  possibility  of  a 

normal  approximation  to  the  exact  posterior  marginal  distribution  of 

(see  section  5) . 

In  passing,  we  note  that  another  attempt  to  approximate  b(YjtR^  by  substl- 
tutlng  the  kernel  of  Inverse  chl-square  densities  ^]^(4^p)  ^nd  ^2^^r^  ^r' 
having  modes  same  as  the  modes  of  <tp  in       and  ^l^^r'  ^j^  ^2^^r^ 

In  (4.2)  was  also  made.    In  this  case. 


and 

_     -  2 

^    r        I  r  1 

Thus,  we  have 

_  ^  +  w  -  1 

«  {X  +  [  i:  (g.  -  gt>^  -  (m  -  2)v]  +  (y.  -  g*)^  -  J}  ^ 

(4.4) 

The  result  of  replacing  f^  and  ^2  ^2  ^"  '^^'^j'?^  ^^^^ 

-JjCv  +  m  -  1) 


b*(Y^|»>  «  exp  t-Sfv'^(Yj  -  gj>^l 


(A  +  [  Z  (gi  -  i*)^  -  (m  -  2)T]  +  ^  (Yj  -  g*>^  -  I) 

(4.5) 
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Examining  (4*5),  we  find  that  b  CYj  U)  is  of  the  same  form  of  bCy^lv**  g) 
in  CA*1)  except  an  extra  term  —  and  that    2  (y    -  yO    is  replaced  by  the 

quantity    I  (g    -  gj    -  (m  -  2)v  *    Again,  as  m,  n  b  (Yjg)  approaches 

±H  ^  ^ 

a  nooial  density* 

The  approximation  of  b(Yjig)  by  b  (Yj|g)  is  in  effect  a  special  case 

of  those  by  bCy^l^  >  g) *    This  is  so  because  if  we  adopt  Jackson's  (1972) 

2  2 
proposed  estimates  of  £(g,  -  g*)    -  (m  -  l)v  for  Ky,.  -  Y*)    and  g*  for  Y* 

2  ^  2 

(in  the  present  context,    Z  (y.  -  y.)    is  estimated  by    Z  (g.  -  g^)    -  (ni  -  2)v 

and  Y*  by  g*)  and  ignore  the  term  v/m  (which  should  be  negligible  even  for 

moderate  m  and  n) ,  we  can  treat  b  (Y^!g)       derived  from  b(Y4lY  ,  g)  * 

Though  seemingly  appealing,  thi3  effort  to  obtain  an  approximation  for 

b(Yj!g)  also  fails*    However,  it  is  a  comfort  to  learn  that  a  normal 

approximation  to  b(Yj|g)  has  been  found  satisfactory*    'this  approxim^-ition 

is  discussed  in  the  next  section* 
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5t    A  Normal  Approximation  to  the  Posterior  Marginal  Distribution  of  Gamma 

Searching  for  an  approximation  to  the  posterior  marginal  distribution 
of  carefully  studied  the  shape  of  its  exact  form^    The  index  of 

skewness  shCFws  that  it  is  only  slightly  skewed  and  that  the  skewness  can 
therefore  be  ignored*    In  general,  the  marginal  distribution  of  given 
g  is  positively  skewed  if  the  observed  g^  is  greater  than  g*,  and  negatively 
skewed  if       <  g»  •    The  exact  ordinates  b(7j|g)  of  its  density  curve  at 
those  points  within  the  range  of  ±  2  standard  deviations  (a^)  from  Its  mean 
(Mj)  in  steps  of  ^5       were  evaluated  by  numerical  integrations*  The 
results  invariably  indicate  a  pattern  of  unitoodality*    The  density  is 
higher  in  the  central  region  around  the  mean  and  decreases  as  moves 
away  from       *    This  suggests  a  good  possibility  of  approximating  this 
density  curve  by  a  normal  curve  except  perhaps  in  the  tails*    Also,  we  recall 
that  indeed  it  has  a  normal  density  as  its  limiting  form*    We,  therefore, 
compared  these  exact  ordinates  b(7^|g)  with  the  corresponding  ordinates  of 
a  normal  curve  whose  mean  and  standard  deviation  coincide  with       and  of 
the  exact  distribution  for       given  g  *    These  comparisons  did  bear  out  our 
conjecture  that  the  normal  approximation  is  a  promising  approach* 

As  an  example,  a  data  set  which  was  the  result  of  a  test  of  12 
items  administered  to  35  children  was  used*    There  were  3,  A,  5,  12,  and 
11  persons  with  8,  9,  10,  11,  and  12  correct  answers,  respectively* 
Columns  2  and  3  of  Table  1  contain  the  ordinates  bCv^lg)  of  the  exact 
distribution  of  ^jlg         t:he  persons  j  having  10  correct  answers,  and 
those  of  the  corresponding  normal  curve*    The  points  7^  ■=  7^  +  ca^,  where  c 
takes  the  values  from  -2*5  to  +2*5  in  steps  of  *5,  were  included*  We 
remark  that  in  making  decisions,  the  relevant  information  is  often 
based  on  the  cumulative  probability  rather  than  the  density  itself* 
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For  this  reason,  columns  4  and  5  present  the  exact  cumulative  probabilities 
ProbCy,      Y_lg)  and  the  corresponding  normal  approximations.    It  is  seen 
that  the  discrepancies  between  the  exact  and  approximate  figures  are  lese 
than  .01»    In  most  practical  applications,  this  accuracy  should  prove  to 
be  entirely  satisfactory. 

'Hie  currently  available  program  MARPRO  provides  the  exact  probability 
ProbU^  >_  TT^Ig)  ^  ProbCy^  ^Y^ls)  as  well  as  its  normal  approximation* 
This  normal  approximation  has  been  found  to  be  very  adequate*    The  differences 
are,  in  fact,  less  than  *005  in  nearly  all  cases.    More  numerical  illustrations 
are  given  in  the  next  section. 

Frequently,  one  is  interested  in  finding  the  100a  percentage  points 
for  gamma.    They  are  difficult  to  evaluate  directly  from  the  exact 
distribution  of  y^|g  .    However,  one  could  find  the  approximate  lOOot 
percentage  point  y^^  for  y^  using  the  unit  normal  curve,  since  the  normal 
approximation  is  usually  expected  to  be  sufficiently  accurate*    For  this 
purpose,  we  now  derive  an  expression  for       |=  Var(yj[g)|  in  terms  of 

P 


[=  Var(y  |g)l 


<^*^  =  Var  (T^lg)  .  (5.1) 


First,  we  find 


Var  [  tl^CYjUp,  S)l 
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(gj  -  S^^^Var  (  lg)l 

(g,  -  g^)^        .  (5.2) 
J 


Applying  equations  (2*5),  (2*6),  and  (5*1),  we  then  have 


0^  =  Var(yj!g)  =  v 


r 

(m  -  Dp*  +  1 


+  (gj  "  g*)    ^*    *  (5*3) 


The  advantage  of  using  the  formula  (5*3)  is  that  In  order  to  find  the  approxi- 
mate 100a  percentage  points  of       for  all  groups,  we  need  only  to  compute 

jL  jL  2  I  ^  jt2 

p    and  0      *    Given  b(<>p|g)  In  equation  (2*2),  p    and  o     are  easy  to  compute 

2 

by  numerical  Integrations,  for  known  m,  n,  X,  v,  and  i:(g   -  g*)  * 

*  *2  ^ 

Now,  If  one  has  P    and  a     available,  the  100a  percentage  point  y^^  for 

y^|g  can  be  obtained  with  the  help  of  a  standard  normal  table*    Thus,  let 

prob(z  i  z^)  =  a  , 

where  z  is  a  standard  normal  varlate,  one  finds 


+  p*(g.  -  gO  +  g. 


such  that 


The  sine-squared  transformation  of  Yq.j  can  then  be  taken  as  the  approximate  100a 

2 

percentage  point  for  ir_j,  viz*,  ir^^  ^  sin    y^^  for  which  prob(ir^  <  ^^^\&)  -  a  * 

it  i2 

Similarly,  knowing  p    and  o    ,  one  could  evaluate  the  approximate  probability 

of  Yj      Y    given  g  from  the  normal  table: 
j  —    o  V 
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ProbCv,  <  Y  |g)      ProbCz  <  z  ) 
1  —    O'  —  o 


■where 


2 


1^ 


Before  leaving  this  discussioDi  we  note  that  one  can  write 


a 


(5-4) 


where 


(5-5) 


In  tabulating  constants  for  the  normal  approximation!  it  turns  out  (Wang,  1973) 

to  be  mote  convenient  to  tabulate  p*^  and  <o*^  than  p*^  and  "J*^  as  a  function 

of  the  prior  and  sample  estimates  of       (x/v  and  2(g.  -  gO  /m,  respectively) 

i 

given  fixed  values  for  m  and  n  •    This  is  so  because  a     is  not  monotone  in 

*2 

the  arguments  but  <o     is  monotone* 
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6»    Numerical  Examples 

in  addition  to  performing  all  the  computations  outlined  In  sections  7. 
and  5,  the  Fortran  program  MARPRO  also  provides  the  joint  modal  estimates 
for  the  gammas*    The  estimates  tt^  and  tt^  of  the  proportions  tt^  based  on 
Yj  and  the  marginal  mean  estimates  Yjt  respectively,  are  both  available  from 
MABFRO  for  comparisons*    Hence,  the  program  MARPRO  Is  recommended  for  analyzing 
m-group  binomial  data  with  equal  n  *    This  program  uses  the  Freeman-Tukey  (1950) 
transformations  for  binomial  data;  l*e*, 

where  x  is  the  observed  number  of  successes*  In  accord  with  thla  transformation 
(6*1),  the  proportions  ir^  can  be  estimated  by: 

.  (1+^)  sln^Yj  .  (6.2) 

(See  Novlck,  Lewis,  and  Jackson,  1973)*    Note  that       Is  also  obtained 

from       by  equation  (6*2)* 

With  the  help  of  this  program,  we  were  able  to  reanalyze  the  data 

presented  In  Tabl^  VI  of  Novlck,  Lewis,  and  Jackson  (1973)*    These  data 

were  collected  for  the  estimation  of  item  difficulties  for  six  social  studies 

Items*    For  a  comparable  analysis,  we  chose  to  set  V  «  8  and  t  >*  6  (which 

V  -  2 

Is  equivalent  to  let  X  «         ^       -  *214  In  the  current  program)*    In  Table  2, 


n 


estimates  of  these  Item  difficulties  ir^  baf^ed  on  Yj  ^nd  Yj  were  presented. 

For  the  joint  modal  estimates,  both  the  present  results  (labeled  FT)  and 

those  of  the  previous  analysis  (labeled  B,  following  the  cited  source)  were 

given*    Notice  that  for  some  groups  slight  discrepancies  between  these  two 

values  were  found  due  to  il**ferent  transformations  employed  [in  Novlck, 

Lewis,  and  Jackson,  Anscombft's  (1948)  transformations  were  taken]*  Both 
it 

p(«  .8856)  and  p  ("  *8906)  are  quite  big  because  of  the  fairly  large  sample 
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sizes  (n  -  57)  for  these  data.    Accordingly,  no  substantial  regressions  of 
individual       estimates  towards  a  common  value  (n.  corresponding  to  y*)  were 
expected*    Similarly,  for  this  large  n,  the  marginal  mean  estimates  do  not 
differ  significantly  from  the  joint  modal  estimates* 

In  the  present  analysis,  the  probability  that  the  item  difficulty  Tt^  is 
greater  than  some  specified  value       was  computed  from  the  marginal  distribution 
of       *    The  exact  (posterior)  probabilities  and  their  normal  approximations 
(given  in  parenthesis),  for       *  *95(***05)*50,  are  presented  in  Table  3* 
The  normal  approximations  were  excellent  in  this  case*    Thus,  having  the 
marginal  distributions  available,  we  can  now  make  explicit  probability 
statements  about  the  item  difficulties  of  these  six  items*    For  example,  one 
finds  the  probability  that  the  item  difficulty  of  item  1  is  greater  than  ,85 
is  *9616.    These  statements  should  prove  to  be  useful  in  selecting  items  for 
a  test.    It  is  interesting  to  note  that  the  posterior  distribution  for  item 
one  assigns  a  probability  of  only  *18  to  the  event       >  .95,  even  though  the 
observed  proportion  was  *9474.    On  the  other  hand,  the  probability  that 
iTj^  <  .90  is  .28*    Thus,  we  see  that  the  posterior  distribution  of  Tt^  is  highly 
asymmetric,  (note  that  posterior  marginal  mean  estimate  of       is  *925^  in 
contrast  to  the  posterior  distribution  of  Yj  which  is  quite  symmetric* 
Fc>r  reference,  the  descriptive  statistics  (mean,  standard  deviation,  and 
index  of  skewness)  for  the  marginal  distributions  of  gamma  are  also 
provided  in  the  same  table*    We  noted  earlier  that  a  uniform  distribution 
on       had  been  assumed  in  the  derivation  when  In  fact       is  restricted  to 
the  range  zero  to  ir/v  *    To  demonstrate  that  this  does  not  materially 
affect  the  analysis  we  numerically  computed,  the  aposteriori  probability 
that       lie  in  the  range  0  to  In  for  each  of  the  data  sets  presented  here. 
In  each  instance  that  probability  was  unity  with  an  accuracy  of  10  . 
The  point,  of  course.  Is  that  provided  m  is  moderate  the  prior  distribution 
on  Up  will  have  little  effect  on  the  results  of  the  analysis* 
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To  Illustrate  the  differences  between  the  marginal  mean  and  the  joint 

modal  estimates  when  the  sample  size  n  Is  smaller,  our  second  example  Invjlves 

some  artificial  data  sets.    Binomial  data  of  m  groups  were  randomly  generated 

from  ^  normal  distribution  of  gamma  with  mean  p„  °  sin    VU    and  variance  4>n  * 

r  ^  1 

First,  m  values  of      were  generated  by  randomly  sampling  Yj  from  the 
specified  normal  distribution.    These  g^  were  mapped  Into  p^  by  sine-squared 
transformations  (p^  »  sin  g^)^    Then  the  nearest  Integers  of  np^  were  taken 

as  the  observed  number  of  successes  x,  to  be  analyzed  by  MARFRO.    All  the 

3 

analyses  reported  hereafter  adopt  v  =  8  and  A  -  .25  (which  Is  equivalent  to 
a  value  of  t  *  5)  for  the  prior  Inverse  chl-square  density  of  . 

We  have  thus  generated  nine  sets  (for  m     10(5)50  and  n  »  8)  of  data. 
The  values  Mp  ■  1.1731,  which  matches  an  average  of  the  proportions       «  .85, 
and  4>p  =  .029  which  happens  to  be  (4n  +  2)  ^  for  n     8,  were  used.    Each  data 
set  was  processed  by  MARPRO.    The  results  demonstrate  consistent  patterns 
for  all  data  sets  and  with  only  minor  differences  for  the  different  values 
of  m  .    We,  therefore,  chose  to  report  only  the  results  for  m  =  10,  20, 
30,  40,  and  50. 

In  Table  4,  the  estimates  iy  based  on  the  marginal  mean  estimates 

and  TT^t  (given  In  parenthesis)  based  on  the  joint  modal  estimates  Yj  were 

presented.    Since  there  were  many  groups  having  the  same  observed  number 

of  Successes       and  thus,  sharing  the  same  estimates  of  n^,  we  present 

these  estimates       ^nd       for  different  values  of  x  Instead  of  for 

each  group.    The  analyses  of  these  generated  data  Invariably  result  In 

it 

significantly  bigger  values  for  P    than  p,  so  that  the  general  conclusion 
(3.3)  follows.    It  Is  also  seen  that  there  are  substantial  differences 
between  tt^  and  . 


26 


As  Che  number  of  groups  (m)  Increases  (for  fixed  n),  both  p  and  p  decrease 

This  means  that  the  estimates  of       t*re  more  regressed  when  more  groups  are 

used.    However,  one  should  bear  In  mind  that  sampling  fluctuations  In  these 

generated  data  result  in  small  variances  In  this  trend.    It  waa  also  found 

ft 

that  the  decreasing  rate  of  p    as  m  Increases  Is  not  as  high  as  that  of  p  . 
This  confirms  the  expectation  that  the  Joint  modal  estimates  are  subject  to 
more  Influences  from  other  groups*    Ori  the  other  hand,  the  marginal  mean 
estimates,  associated  with  a  squared^^error  loss  for  each  group  separately, 
place  mor^.  emphasis  on  the  Individual  observations.    Thus,  they  are  less 
jiffected  by  the  Inclusion  of  more  groups. 

For  the  marginal  probabilities  ProbU^  >  ir^ls),  we  arbitrarily  selected 
those  for  groups  with  the  number  of  successes  ^ j  "  ^        ^       bft  reported  In 

Tables  5  and  6.    The  values  of  it^  from  .70  to  .95  by  step?  of  •OS  were 

Included  In  the  tables*    The  normal  approximations  are  again  sufficiently 

precise*    The  trend  of  Increases  in  the  probabilities  as  m  Increases  Is 

consistent  with  the  results  In  Table  A.    Since  the  marginal  distributions 

are  relatively  stable  w»r»t»  the  size  of  m,  we  suggest  that  the  observed 

differences  are  largely  due  to  sampling  fluctuations  In  our  generated  data. 

In  passing,  wg  note  that  other  data  sets  generated  In  the  same  way 

described  earlier  for  n  =  6  and  virious  sizes  of  m  have  also  been  analyzed. 

The  results  reveal  the  same  patterns  found  In  the  above  example. 

Oiir  last  example  used  the  result  of  a  12-item  test  administered  to 

35  children.    The  outcome  was  that  11  persons  scored  perfectly,  12  persons 

missed  only  one  Item,  and  3,  4,  and  5  per^^ons  gave  correct  answers  to  8^  9, 

and  10  Items,  respectively.    The  estimates       and       and  the  posterior 

ir-cirglnal  probabilltlefi  prob(:r^  >  ir^lg)  were  prt*Bcnt^d  In  Table  7.  A^;aln» 

tM>U!;ldevable  dffl'erences  between  Sj  m\\       were  recorded.    TUg  postorlor 
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probabilities  enable  us  to  reach  a  more  specific  judgment  on  the  individual's 

performance.    For  example,  a  person  j  having  8  correct  answers  (p^  =  .6667) 

in  the  test  is  considered  to  have  an  ability  greater  than  .55  with  high 

certainty  [prob(iTj  >  .55|g)  .9912]. 

The  analysis  in  Table  7  demonstrates  the  force  of  the  Bayesian  m-group 

method.    Consider  a  class  performance  as  indicated  in  the  data  for  Table  7 

and  a  situation  in  which  a  mastery  level  of       »  .85  seemed  appropriate. 

We  note  that  a  person  answering  10  items  correctly  has  a  p^  ''score**  of 

.833,  and  hence,  has  failed  the  it    criterion  value  of  .85.    As  a  result, 

o 

we  would  not  pass  the  person.    The  Bayesian  analysis,  however,  yields  a 
different  picture.    First,  the  point  estimates  of  his  it  are  .8829  and  .8657 
relative  to  joint  zero-one  loss  and  either  joint  or  component  squared-error 
loss,  respectively.    Thus,  on  an  informal  basis,  we  would  probably  decide 
to  pass  the  person.    Secondly,  the  probability  that  his  score  is  at  least 
.85  is  .5082.    Therefore,  with  roughly  equal  losses  associated  with  false 
positives  and  false  negatives  it  would  essentially  be  a  toss-up  as  to  whether 
he  was  passed  or  not. 

In  passing,  we  also  note  that  for  a  person  j  with  11  correct  answers, 
the  joint  estimate  tf^  is  identical  to  the  marginal  estimate       (=  .9036). 
This  is  so  because,  for  this  person,  his  observed  g  score  (g^  "  1.2288)  is 
equal  to  the  average  g  score  over  all  persons  (g.  =  1.2287),    It  is  also 
clear  that,  from  equation  (2.9),  the  posterior  marginal  distribution  of  the 
corresponding       is  symmetric  (i.e.,  the  coefficient  of  skewness  5.  *  0) . 

Finally,  posterior  conditional  means  and  standard  deviations  of 

given  the  marginal  mode       of  ^^jg  were  computed  for  the  data  of  the  six 

social  studies  items  and  some  of  the  randomly  generated  data  sets  (see  Table 

8),    These  conditioual  mean  estimates  of  Y.  were  compared  with  their  marginal 

1 
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mean  estimates  to  provide  some  Idea  about  how  large  the  values  of  m  and  n 
would  warrant  the  use  of  conditional  estimates  [which  are  of  much  simpler 
form  as  Indicated  by  equation  (2*1)]  as  approximations  to  the  posterior 
estimates  of       .    It  appears  that  for  m  ■>=  SO  and  n     30|  the  conditional 
mean  estimates  CCv^j^pt  g)         standard  deviations  <^(Yj|^p|g)  are  reasonably 
close  to  their  marginal  statistics  ^(Vjlg)  and'0(Yj|g)*    Note  that 
referring  to  equation  (2.2)^  the  marginal  mode  |-  of  K  given  g  can  be 
obtained  by  solving  the  following  cubic  equation  for  1^^: 

(m  +  V  +  1)  'tj^  +  [(in  +  2v  +  3)v  -  -  g.)^  -  A) 

+  [(v  +  2)v^  -  2Xv)  Ij,  -  Xv^  -  0      .  (6.3) 
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7*    Summary  and  Conclusions 

The  knowledge  of  the  posterior  marginal  distributions  of 
gaonitad  should  3ld  In  making  our  decision  when  It  Is  concerned  about 

Individual  persons*    There  Is  little  doubt  that  the  normal  approximations 
to  these  marginal  distributions  are  very  successful,  judging  from 
comparisons  with  the  exacf  probabilities  obtained  by  Integrations*  Thus, 
we  recall  from  section  5  that  once  p    «       .    (  t — rrr   g)  and  o  « 


Var  ( 


g)  are  computed  by  Integrations,  the  Interesting  descriptive 


Statistics  (mean»  standard  deviation)  for       given  g  are  readily  available* 
*  *2 

Moreover,  given  p    and  0    »  the  relevant  probabilities  for  making 
Individual  decisions  and  the  percentage  points  for       (or  ttJ  given  g  can  be 
«fttlBf«ctorlly  approximated  using  a  standard  normal  table* 
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Table  1:    An  Example  of  the  Posterior  Densities  and  Cumulative 
Probabilities  for       given  g 


c 

Posterior  Density 

Cumulative  Probability 

Exact 

Normal  Approxlmaclon 

Exact 

Normal  Approximation 

-2.5 

.1834 

.1736 

.0071 

.0062 

-2.0 

.5346 

.5346 

.0239 

.0228 

-1.5 

.  UQOO 

-1.0 

2.3558 

2.3961 

.1575 

.1587 

-  .5 

3.4708 

3.4863 

.3058 

.3085 

0.0 

3.9806 

3.9505 

.4976 

.5000 

.5 

3.5292 

3.4863 

.6911 

.6915 

1.0 

2.4118 

2.3961 

.8426 

.8413 

1.5 

1.2726 

1.2825 

.9345 

.9332 

2.0 

.5215 

.5346 

.9779 

.9772 

2.5 

.1676 

.1736 

.9939 

.9938 

The  prior  distribution  of       In  this  analysis  Is  an  Inverse  chl- 


square  with  d.f.  v  »  8  and  paraineter  A  =  .25.    There  are  35  persons^ 
12  observations  In  each.    The  number  of  persons  having  8>  9>  10>  11^ 
and  12  successes  are>  respectively^  3>  4>  5>  12>  and  11.    The  value  g 
corresponding  to  10  successes  Is  g,  =  1.1187  and  the  mean  of  g,  over 
35  persons  Is  g.  ■  1,2287.    The  weight  for  the  modal  estimates  of  v 
Is  p  ■  .2757  and  the  weight  for  the  marginal  mean  estimates  Is 
p    ■  .4920.    The  descriptive  statistics  of  the  distribution  for 
given  g  are:  "  1'1746,       =  [Var(v^[g)]^  «  .1010  and 

5j  »  coefficient  of  skewness  =  -.0035. 


Table  2:    An  Analysis  of  Item  Difficulties  for  Six  Social  Studies  Iteios 


Item 
Number 

n 

Joint  Est.( 
FT 

B 

Marginal  Est. 

1 

57 

.947 

.924 

.922 

.925 

2 

57 

.386 

.423 

.423 

.421 

3 

57 

.526 

.546 

.546 

.546 

4 

57 

.842 

.825 

.823 

.825 

5 

57 

.772 

.762 

.761 

.762 

6 

57 

.614 

.623 

.622 

'  .623 

Prior  distribution  of  <tp:  v  «  8>  t  ^  6  (equivalent ly>  X  =  -214); 
P  -  .8856  and  p    ■  .8906. 


Table  3;    Posterior  Probabilities  prob(iij  >  Tr^)g)  for  the 
Six  Social  Studies  Items 
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I terns 
0  X 

prob(ii 

1 

A 
** 

e 

0 

.95 

.  loUZ 

(.1803) 

.... 

(.0005) 

U  f 

(0.) 

.90 

71  Q  ^ 
.  /  I?  J 

(.7200) 

(.0369) 

(.0012) 

.85 

(.9611) 

(.2789) 

(.0346) 

(0.) 

.80 

(.997A) 

~  ~  "  " 

U  . 

(0.) 

.0  7j7 
(.6762) 

(.2207) 

i\f\A€y 

(.0008) 

.75 

.9999 
(.9999) 

.UUU3 

(.0003) 

»92U3 
(*9201) 

C  711 

(.5725) 

.U13!) 
(.0136) 

.70 

l.U 

(1.0) 

U. 

(0.) 

(.0056) 

.9o93 
(.9890) 

.8585 
(.8584) 

.0937 
(.0939) 

.65 

.0001 
(.0002) 

Ai  i  o 

.OAAo 
(.0A52) 

A  rt  rt 

.9992 
(.9991) 

.9728 
(.9727) 

.3198 
(.3196) 

.60 

.002A 
(.0026) 

.1903 
(.1902) 

1.0 
(1.0) 

.9970 
(.9970) 

,6379 
(.6375) 

.0220 
(.0225) 

.470*0 
(.A692) 

.9998 
(.9993) 

.8760 
(.8760) 

.50 

.1101 
(.1104) 

.7629 
(.7626) 

.9999 
(.9999) 

.9743 
(.9745) 

.9250 

.A211 

.5455 

.8254 

.7622 

.6227 

.9A74 

.  3860 

.5263 

.7719 

.6140 

1.3232 

.6724 

.8112 

1.1543 

1.0674 

.8984 

1.2865 

.7069 

.8306 

1.1360 

1.0587 

.9083 

0(Yj|g) 

.06A3 

.0641 

.0633 

.0632 

.0629 

.0630 

6. 

3 

.0011 

-.0010 

-.CW)07 

.0007 

.0004 

-.0004 

The  exact  probabilities  were  obtained  by  numerical  integrations.  Their 
corresponding  normal  approximations  were  given  in  parentheses.  Those 
probabilities  less  than  .0001  and  greater  than  ,9999  were  regarded  as  0 
and  1(  respectively,  p    Is  the  observed  sampled  proportion  and  6    is  the 
Index  of  skewness  of  the  conditional  distribution  of  y^jn  .    For  this 
set  oC  data,  g.  «  .9878  and  i:(g    -  r.)^  =  ,2852. 
 270 


Table  4;    Analyses  of  Data  Sets  Randomly  Generated 
estimates  of 
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10 

20 

30 

40 

50 

3 

.632 





.672 

(p.  -  .375) 

J 

—  u 

(.713) 

w  <— _  — 

(.776) 

4 



.685 



.718 

.715 

(P.  '  .500) 
J 

____ 

(.743) 

(.715) 

(.791) 

5 

.720 

.735 

.735 

.759 

.756 

(P.  -  .625) 

J 

(.740) 

(.772) 

(.772) 

(.802) 

(.805) 

6 

.772 

.786 

.781 

.800 

.798 

(p.  -  .750) 

J 

(.779) 

(.802) 

(.794) 

(.819) 

(.820) 

7 

.828 

.841 

.829 

.844 

.843 

(P^  -  .875) 

(.821) 

(.835) 

(.819) 

(.839) 

(.837) 

g 

.  923 

905 

913 

(p,  -  1.000) 

(.889) 

(.888) 

(.861) 

(.871) 

(.865) 

p* 

.4620 

.4603 

.4068 

.3789 

.3853 

p 

(.3518) 

(.2792) 

(.2079) 

(.1679) 

(.1446) 

g. 

1.0853 

1.1146 

1.0950 

1.1260 

1.1238 

2(gi  -  g.)^ 
1 

.1723 

.7388 

.8257 

1.0257 

1.5945 

These  data  were  randomly  generated  from  a  normal  distribution  for  y  with 
mean  Pp  -  1.1731  (p^  -  .85)  and       =  .029.    The  number  of  observations  in 
each  group  is  n  »  8.    The  present  analyses  adopt  v  =  8^  t  =  5 
(equivalently  X  ^  .25)  for  the  prior  inverse  chi-square  density  of  . 
Marginal  estimates       and  joint  estimates'?^  (given  in  parentheses) 
are  presented  here.    Blank  entries  indicate  there  are  no  values 
of  the  corresponding  x    being  sampled,    p-  -  x  /n  Is  the  observed 
sample  proportion  of  group  j  .    Note  that  p->^  (.3853)  for  m  »  50  is 
larger  than  p^^  (.3789)  for  m  «  40  due  to  sampling  fluctuations.  The 
generated  data  for  m  =  50  has  a  bigger  mean  squared  deviations  of 
g  [|(g^  -  g.)  /m]  than  that  of  the^a*;^  for  m  ^  40  (,03189  as  compared 
to  .02564).  ^  '  ^ 
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Table  5:    Posterior  Marginal  Distributions  of  Yj  for  Groups  with 
5  Successes  in  8  Trials  for  the  Five  Generated  Data  Sets 


prob(Tr    >  ii^lg) 


10 

20 

30 

40 

50 

.95 

.0024 
(.0027) 

.0028 
(.0032) 

.0016 
(.0017) 

.0022 
(.0025) 

.0021 
(.0024) 

.90 

.0207 
(.0222) 

.0248 
(.0268) 

.0179 
(.0192) 

.0254 
(.0273) 

.0244 
(.0261) 

.85 

.0783 
(.0809) 

.0933 
(.0961) 

.0783 
(.6809) 

.1077 
(.1102) 

.1037 
(.1062) 

.80 

.1912 
(.1923) 

.2233 
(.2236) 

.2063 
(.2072) 

.2683 
(.2677) 

.2604 
(.2600) 

.75 

.3514 
(.3590) 

.3991 
(.3952) 

.3906 
(.3876) 

.4773 
(.4725) 

.4667 
(.4623) 

.70 

.5299 
(.5248) 

.5839 
(.5778) 

.5886 
(.5832) 

.6773 
(.6720) 

.6672 
(.6621) 

.720 

.735 

.735 

.759 

.756 

.9989 
.1245 
-.0091 

1.0149 
.1213 
-.0099 

1.0149 
.1130 
-.0080 

1.0397 
.1089 
-.0083 

1.0369 
.1093 
-.0079 

For  these  groups,       =  .625  and       =  .8982.    The  exact  probabilities 
prob(ff^  >  I'glg)        the  corresponding  normal  approximations  (in 
parentheses)  are  presented. 
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Table  6:    Posterior  Marginal  Distributions  of       for  Groups 
with  7  Successes  in  the  Five  Generated  Data  Sets 


prob(Ttj  >  %U> 


m 

^o\^ 

10 

20 

30 

40 

50 

.95 

.0335 
^.0325} 

.0390 
t  *  0  38 1|  J 

.0230 

.0273 
(.0267} 

.0271 
^.0265} 

.90 

.1421 

.1652 

.1228 

•  1477 

.1463 
\*  m  fj/ 

.85 

.3238 
(.3268) 

.3678 
(.3701) 

.3137 
(.3165) 

.3673 
(.3693) 

.3638 
(.3658) 

.80 

.5327 
(.5342) 

.5866 
(.5867) 

.5432 
(.5441) 

.6102 
(.6096) 

.6056 
(.6051) 

.75 

.7171 
(.7163) 

.7658 
(.7641) 

.7421 
(.7407) 

.8000 
(.7981) 

.7959 
(.7941) 

.70 

.8493 
(.8476) 

.9841 
(.8825) 

.8758 
(.8740) 

.9135 
(.9123) 

.9109 
(.9097) 

.828 

.841 

.829 

.844 

.843 

1.1177 

1.1334 

1.1196 

1.1371 

1.1360 

.1234 

.1198 

.1121 

.1077 

.1082 

.0034 

.0019 

.0025 

.0011 

.0011 

For  these  groups^  p*  =  .875  and  g*  =  1*1554. 
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Table  7:    Analyses  of  a  Data  Set  Obtained  from  a  12-item  Test  Given 
to  35  Children 


X. 

J 


84  = 


8 

.6667 
.9423 


X. 

J 


84  = 


9 

.7500 
1.0262 


X.  = 


2 
8^ 


10 

.8333 
1.1187 


X.  = 


P4  = 


11 

.9167 
1.2288 


X.  = 
J 


12 

1.0000 
1.4303 


.95 
.90 
.85 
.80 
.75 
.70 
.65 
.60 
.55 


.0052(, 
.0557(, 
.2035(, 
.4304(, 
.6588(, 
.8276(, 
.9258(, 
.9725(, 
.9912(, 


0062) 
0585) 
2035) 
4253) 
6534) 
8262) 
9276) 
9750) 
9929) 


.0156(.0168) 
.1174(.1192) 
.3348(.3327) 
.5898(.5853) 
.7917(.7894) 
.9115(.9123) 
.9680(.9699) 
.9901(.9915) 
~  (    ~  ) 


.0444 (.0455) 
.2298(.2304) 
.5082 (.5058) 
.7502(.7478) 
.8966(.8964) 
.9642(.9653) 
.9895(.9905) 
.9973(.9979) 
"  (    --  ) 


.1224(.1234) 
.4193(.4199) 
.7110(.7097) 
.8873(.8864) 
.9642(.9643) 
.9905(.9905) 
~  (    —  ) 
~  (    ~  ) 
(    -  )- 


.4?J5(.4321) 
.7808 (.780 7) 
.9379(.5359) 
.9861C.9850) 
.9974(.9971) 
~  (    —  ) 

—  (    -  ) 

—  (  -  ) 
--  (    --  ) 


Estimates  of  Proportions  and  Descriptive  Statistics 


c":(Yjl8) 

'3 ' 


(.8468) 
.7961 

1.0878 
.1029 

-.0089 


(.8644) 
.8305 

1.1291 
.1018 

-.0064 


(.8829) 
.8657 

1.1746 
.1010 

-.0035 


(.',036) 

.9036 
1.2287  ' 

.1007 

.0000 


(.9376) 
.9606 

1.3279 
.1018 
.0063 


For  these  data,  g.  =  1.2287,  IKg    -  g.)    =  .9175.    Prior  distribution  of 

i  * 

^^i    V  =>  8,  t  =  5  (equivalently,  X  =  .25).    p  =  .2757  and  p    =  .4920.  The 

figures  in  parentheses  are  normal  approximations  to  prob(ii.  >  u  |g).  Those 

J  —    o  = 

cumulative  probabilities  greater  than  .999  were  omitted  in  the  table. 


8  prob(.8861  <  <  1.2895)  = 

9  prob(.9296  <  <  1.3286)  = 

10  prob(.9766  <  <  1.3726)  ■■ 

11  prob(1.03l3  <  <  1.4261) 

12  prob(1.1284  <  Yj  <  1.5274) 


.95  or  prob(.6000  <  <  .9229)  ^  .95 

.95  or  prob(.6422  ^  ^  .9425)  .95 

.95  or  p^obi.6866  <  <  .9612)  =  .95 

'  .95  or  prob(.7361  <  <  .9792)  ^  .95 

■  .95  or  prob(.8167  <  <  .9981)  =  .95 
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Table  8:    Conditional  Distributions  of  Yj  Given  the 
Marginal  Mode  !«  of  <|>p|g 


I.    The  Six  Social  Studies  Items  (m  »  6,  n  "  57) 
$p  =  .03004 


.9A7A 

.3860 

.5263 

.8A21 

.7719 

.61A0 

.9250 

.A  211 

.5A55 

.925A 

.  7622 

.6227 

l8> 

(.9219) 

(.A265) 

(.5A86) 

(.8233) 

(.7610) 

(.62A2) 

1.2865 

.7069 

.8306 

1.1360 

1.0857 

.9083 

|^P>  g) 

(1.2808) 

(.7122) 

(.8336) 

(1.1332) 

(1.057A) 

(.9097) 

l|> 

.06A3 

.06A1 

.0633 

.06:j2 

.0629 

.0630 

lip,  g>* 

(.062A) 

(.0624) 

(.062A) 

(.062A) 

(.062A) 

(.062A) 

II.    Randomly  Generated  Data  (m  «  50,  n  =  8) 
Ip  -  .01652 


*3750 

.5000 

.6250 

1  1 

.7500 

.8750 

1.000 

h  _ 

.6717 

.7U7 

.7561 

.7977 

.8427 

.9129 

(.6833) 

(.7231) 

(.761A) 

(.8002) 

(.8421) 

(.9081) 

.9A99 

.993A 

1.0369 

1.0829 

1.1360 

1.2306 

(.9615) 

(1.0021) 

(1.0A27) 

(1.0856) 

(1.1352) 

(1.2234) 

o(Yjl|) 

.  1125 

.1106 

.1093 

.108A 

.1082 

.1098 

o(Yjllp»  §)* 

(.10A7) 

(.10A7) 

(.10A7) 

(.1047) 

(.1047) 

(.1047) 
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III.    Randomly  Generated  Data  (m  «  50,  n  «  30) 
?P  -  .00924 


.66fi7 

.7667 

.8000 

.8333 

.8667 

.9000 

.9333 

.9667 

.7535 

.8031 

.8200 

.8374 

.8552 

.8740 

.8941 

.9167 

(.7564) 

(.3045) 

(.8209) 

(.8377) 

(.8550) 

(.8733) 

(.8928) 

(.9150) 

l> 

1.0464 

1. 1048 

1.1259 

1.1483 

1.1723 

1.1987 

1.2289 

1.2660 

(1.0498) 

(1.1065) 

(1.1270) 

(1.1487) 

(1.1720) 

(1.1977) 

(1.2270) 

(1.2630] 

l> 

.0685 

.0677 

.0675 

.0675 

.0674 

.0675 

.0678 

.0683 

(.0665) 

(.0665) 

(.0665) 

(.0665) 

(.0665) 

(.0665) 

(.0663) 

(.0665] 

Data  eets  I  and  II  were  used  In  Tables  2  and  4|  respectively.    Data  set 

III  was  generated  specifically  for  this  table.    The  sample  atatisticG  for 

2 

Data  set  III  are  g.  «  1.1625  and  Z(g,  -  g.)    «  .2927. 


*The  conditional  standard  deviations  of       given  4»p  and  g  are  seme  for  all 
groups. 
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Before  concluding  our  discussion  of  the  estimation  of  proportions  in 
m  groups,  we  shall  briefly  consider  the  effect  of  assuming  a  proper  prior 
distribution  for  u^,  in  place  of  the  uniform  distribution  used  up  until 
now.    The  form  of  the  prior  is  specified,  if  we  restrict  ourselves  to 
natural  conjugate  densities,  by 

bdipl^p)  e^p[-h  k  4^p\yp  -  h)^l  ,  (1) 

where  h  is  the  prior  mean  for  Vp  and  k  tha  "prior  sample  size"  associated 
with  our  knowledge  of       .    Combining  Equation  (1)  with  bCyj^ip,  ^p)  and 
b(*l>p),  we  obtain 

b(Y,  Vp,  <tp)  -  <^p^''  +  m  +  3)  exp{-i5<tp^2(Y.  -  Up)^  +  k(Pp  -  h)^  +  X]}  . 

(2) 


*This  note  is  a  Technical  Supplement  to  ACT  Technical  Bulletin  Ko.  13. 
The  material  coatained  here  should  be  considered  as  inserted  prior  to  the 
concluding  section  of  that  Bulletin. 
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Kow»  with  some  rearranging  of  terms,  we  may  write 


ERIC 


2 

ft   I  kh  +  my*  \ 


+  <^p^[X  +  Ky^  -  yO^  +  1^  (h  -  YO^l 


Thus»  i£  we  integrate  Equation  (2)  w.r.t.  Pj,,  we  obtain 

b(Y,  V  "  ^'r''^''  exp{-!i$-^X  +  Ky.  -  Y.)^  +  ^  (h  -  yO^]}  . 

Further  integration^  this  time  w.r.t*  y^^lds 

b(Y)  «  [X  +  2(Y,  -  yO'  +  <h  -  Y.)']-^-^^ 

[v  +  (y  -  hl)'A(Y  -  hl))"^^''  ^       ,  (3) 

where  1  is  the  vector  of  order  nt  all  of  whose  elements  are  unity  and  A  is 

the  m  X  m  matrix  with  diagonal  elements  (     ^  ^         )      and  of£*-diagonal 

K  +  m  A 

elements  (      — ■  )  t  *         other  words>  ue  have  shown  that  tt^e  unconditional 
K  +  m  A 

prior  distribution  for  ;^  is  multivariate  t,  with  v  degi'oes  of  freedom^ 

mean  hl>  and  covariance  matrix  (  ^  ^  2  *         particular^  this  implies 

that  the  marginal  prior  density  of  any       is  univariate  t>  with  v  degrees 

k  +  1  X 

of  freedotn»  mean  h>  and  variance  {  )  """2  *  P>^t^vlded  k  is  greater 

than  zero*    We  note  that  if  k  «  0^  the  joint  density  b(Y)  in  Equation  (3) 

11'  ^ 

becomes  improper  because  the  inverse  of  A  =1  1    -  ™—  does  not  exist* 

mm 

Novick>  Lewis^  and  Jackson  (1973)  discuss  the  possibility  of  interrogating 
an  investigator  about  his  prior  beliefs  concerning  tt^^  where  1  ha^  been 
arbitrarily  selected*    One  of  their  suggestions  is  to  approximate  the^e 

28i 


beliefs  with  a  beta  density*    it  we  Interpret  the  parametert^  oi  the  density 

so  obtained       the  numbers  o£  "prior  successes'*  and  '*prior  failures'^ 

respectively,  then  the  suw  o£  the  parameters  gives  the  "pv±r-r  sample  size", 

t,  and  the  mean  o£  the  distribution  is  the  "prior  proportion  o£  successes", 

M  -    From  these  two  values,  Novick,  Lewis,  and  Jackson  (1973)  obtain  approximate 

-1  / — 

expressions  £or  the  mean  and  variance  of       ^  sin    vir^,  namely 


and 

1 


Var(YJ  = 


1'      4(t  +  1)  * 

If  we  now  equate  these  values  to  the  mean  and  variance  for  found  above, 
we  have  expressions  for  h  and  for  X 

h  =  sin""^*^r  (A) 

and 


4(k  +  l)(t  +  1) 


Novick,  Lewis,  and  Jackson  (1973)  have  argued  that  v  =  8  will,  in  many 
cases,  be  a  reasonable  specification  o£  the  prior  degrees  of  freedom  Cor 

.    Tf  we  accept  this  value,  then  our  only  remaining  task  is  to  specify  k, 
the  "prior  sample  size"  £or  lij,  .    It  is  tempting,  and  may  in  some  cases  be 
reasonable,  to  assume  that  our  prior  knowledge  ot       and  of  ^j.  come  from 
essentially  the  sa!i.^  sources  and  so  could  be  associated  with  a  single 
hypothetical  prior  sample^    This  would  allow  us  to  equate  k  -  1  and  v, 
giving  a  value  o£  k  =  9  in  the  present  circumstances*    In  many  cas^s, 
liowever,  when  we  have  selected  our  groups  (or  individuals)  to  be  quite 
similar,  our  knowledge  concerning       may  be  greater  than  our  knowledge^  of 


lij.  .    This  would  suggest  taking  k  <  v  +  1,    Working  with  an  improper  prior 
for  )ip  represents,  in  effect,  the  extreme  situation  where  k  =  0.    If  we 
were  to  work  with  k  =  5,  for  instance.  Equation  (5)  would  i;educe  to 


X  = 


A(t  +  1)  ' 

which  may  often  be  a  reasonable  assignment. 

Once  values  have  been  supplied  for  h,  k,  X,  and  v,  we  can  work 
directly  with  the  posterior  distribution  for  y»  which  is  proportional  to 
the  product  of  the  likelihood  )l(Y|g)  and  the  prior  density  of  Y,  given 
in  Equation  (3) : 

(6) 

If  we  take  derivatives  with  respect  to  each       and  set  the  results  equal 
to  zero,  we  obtair  the  following  equations  for  the  joint  posterior  mode 
of  y: 

/ 

Y,  =  -V  ^  .  (7) 


where 


and 


kh  +  mY* 
"    k  +  tn 


These  equations  are  closely  related  to  Equation  (3*1),  Section  3,  ot  the 
main  text;  the  solution  obtained  with  an  improper  prior  for  Pp  .  Indeed, 
if  h  "  Y-  or  if  k  *  0,  the  two  results  are  Identical  except  for  a  difference 
of  unity  In  the  denominator  of  *p  *    At  a  practical  level,  making  use  of  oar 
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prior  knowledge  about  Mp  increases  the  effective  number  of  groups  in  the 
study;  this  will  be  particularly  important  in  cases  where  m  is  relatively 
small  (say>  between  5  and  15)*    On  the  other  hand,  for  larger  m  or  in  cases 
where  the  prior  specification  closely  agrees  wit\  the  sample  results,  there 
will  be  little  to  choose  between  proper  and  improper  priors  for  * 
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by 
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1,    Oeneral  Descriptions 

For  estimation  of  proportions  in  m  groups,  Kovick,  Lewis,  and  Jackson 

(1973)  have  developed  a  Bayesian  Model  II  solution  which  provides  posterior 

joint  modal  estimates  Y    of  the  transformed  proportions  Y.  =  sln""^*^,   .  The 

J  3  3 

values       (j  =  ^)  f  the  sine-squared  transformations  of  Y^. suitably 

corrected  by  a  factor  depending  on  sample  sizes  n^^  were  then  taken  as  the 

Bayesian  joint  estimates  of  the  group  proportions       .    These  joint 

estimates  are  useful  in  making  joint  decisions  for  m  gro':ps. 

To  aid  in  making  separate  decisions  on  individual  groups,  the  posterior 

marginal  distributions  of  Y^,  for  the  case  of  equal  sample  sizes  n,  hnve 

been  studied  recently  by  Lewis,  Wang,  and  Novick  (1973)*  They 

worked  out  the  posterior  marginal  mean  ostinates       =  cA^ -\^0  of  K  to 

3^3  i 

be 

tij  °  P  8j  +  (1  -  P  )g.     ,  (1) 
where  P    =         (  ^    ^  ^  |g)  and  v  =  (4n  +  2)      .    The  posterior  vatinnccs 

2  r  r 

a_j  of  Yj  were  expressed  as; 
where  c     =  Var^^(  j—^  |g)  . 
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•k  •k2 

Thus>  if  p    and  a      have  been  computed^  one  could  easily  obtain  the 

posterior  means  and  variances  of  Yj  from  formulas  (1)  and  (2). 
*  *2 

To  compute  p    and  ^    >  one  needs  to  know  the  posterior  distribution 
of  (j^j,  .    This  was  given  by  Lewis  #  Wang,  and  Kovick  (1973)  as 

-     -  1 

'  cxp(-is        X)  .  (3) 

A  A2 

The  values  o£  p    and  a     can  then  bo  obtained  by  numerical  integrations; 


and 


where 


*2       *2  *2 
<j     =  (0     -  p      ,  (5) 


w*^  =       ^^i^y  +  v)"^  b(4.j.|g)d4.j.    .  (6) 

A  A2 

Equations  (3)  -  (6)  Inalcate  that  P    and  a     would  vary  for  different 
2  2 

ra>  n>  V,  X  and  S    =  Il(g    -  gO  .    Consequently >  a  complete  set  of  tables 

&  ,  ,  .         *     *  * 

A  A2 

of  p  >  a     for  all  practical  values  of  these  five  parameters  would  require 
a  formidable  volume.    Since  previous  experiences  and  theoretical  findinf^s 
have  suggested  that  v  =  8  was  a  satisfactory  choice  in  most  appllCdtlons 
for  the  prior  distribution  of       (see  Kovlck>  Lewis>  and  Jackson,  1973) >  we  are 
therefore^  content  with  providing  a  sublet  of  the  tables  which  set  v  =  3.  The 
values  m  «  10(5)30(10)80  and  n  =  8(2)30  are  Included  in  the  tablC:? 

A  A2 

presented  here.    For  each  pair  of  (m,  n),  p    and  a     wer*2  computed  for 

2  2 

different  values  of  X/v  and  s    «  S  /m  (prior  and  sample  estimates  of  ^«) r 

g       g  * 

2 

Values  of  X/v>  s    «  .01(.01).05  are  Included  In  the  tables.    For  other 
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values  of  m,  n,  X/v  and  s    within  their  ranges  selected  in  these  tables, 
corresponding  p    and  ^     can  be  approximated  by  Interpolation. 

While  0     is  not  monotone  in  X/v  and  s^,  the  posterior  expectation 

*2         2  -2 
w     of  <tj^(*}>p  +  v)     with  respect  to       is  monotone*    We,  therefore, 

tabulate  p    and  w     Instead  of  p    and  Given  a  prior  estimate  X/v  and 

2  A  *2 

a  sainple  estimate  s    of  <tp,  for  which  P    and  w     are  not  explicitly  tabulated, 

A  A2 

one  can  obtain  P    and  w     by  Interpolation  using  the  ^iven  tables*    The  value 

A2  *2  *2  ^  *2       *2  *2. 

of  cr     then  can  be  found  by  subtracting  p     from  w      (cr     =  w     -  P  )* 

It  may  be  noted  that  these  tables  include  the  size  of  m  and  n  only 

up  to  80  and  30,  respectively.    For  values  of  m  >  80  and  n  >  30,  the  posterior 

conditional  distribution  of       given       [the  posterior  modal  estimates 

of  <tj.  obtained  from  Its  posterior  density  expressed  by  (3)]  and  g  was  found  to 

satisfactorily  approximate  the  posterior  marginal  distribution  of 

given  g  .    This  posterior  conditional  distribution  of  Yj  given  ^j,,  g 

was  shown  to  be  normal  (see  section  2,  I^ewls,  Wang,  and  Novick,  1973)  with 

mean 

ccv.lip,  g)  =    ,  (7) 

J     *     -  9p  +  V 

and  variance  ^  _^ 

v(L  +  m"  v) 

Var(Y.U«,  g)  =   .  (8) 

^    ^    "  <!>p  +  V 

Thus,  for  large  m  and  n,  this  conditional  distribution  provides  au 

approxlinate  basis  for  making  decisions  on  individual  groups.  Having 
A  A2 

made  p    and  <J     available,  the  probabilities  that  a  group  proportion 
TT_j  Is  greater  than  some  criterion  tt^  given  obiiervGd  g  [prob(TT^  >_  f^lg)) 
can  be  obtained  applying  the  normal  approxlmaclon  to  the  postcrltit 
distribution  of  Yj  given  g  discussed  in  (Lewis,  Wang,  and  Novick,  1973)* 
That  is, 
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prob(7r^  >  TT^Ig)  -  prob(Y^  >  Y^|g)  «  prob(z  >  z^)  (9) 
where  y    =  sln'^VTr"  , 

O  0 

r  */  ^         Jl  +  (m  -  l)p*  .   .  .2    *2 1'** 

'~  (10) 

Similarly,  approximate  100a  percentage  points  tt^^  of  tt^  can  be  computed 
with  the  help  of  a  standard  normal  table.    For  example, 

"  '^"^  ^aj  ^^^^ 

where 

^     „  ,    +  ,    [1+  (m-  l)p%  ,     _      .2  *2-]  ^  ^  ^ 

and  z    is  the  100a  percentage  point  of  a  standard  normal  variate. 
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2,  interpolations 


In  practical  applications,  one  would  not  expect  a    of  his  data  be 

S 

2 

exactly  equal  to  the  tabulated  values  (s  *  =  ,01 (,01) .05) *    Likewise,  an 

Investigator  may  have  reason  to  choose  his  prior  X/v  other  than  values 

Included  In  these  tables.    In  these  cases,  approximations  of  p    and  u 

can  be  obtained  by  interpolation  using  available  tabular  points.  For 

A  A  2  2 

Illustrative  purposes,  we  have  computed  p    and  *Jt>     for  a    =  ,0169,  ,0256, 

*0361,  and  ,0484  with  m  »  10,  v  =  8,  X/v  =  ,01  and  n  =  8,  16  by  numerical 

*  *2 

integrations.    These  exact  values  of  p    and  u     are  then  compared  with 
*  *2 

those  (Pj,       )  obtained  by  simple  linear  interpolation.    The  table 
presented  below  shows  that  the  discrepancies  between  Interpolated  and 
exact  values  are  negligible, 

A  ^2 

Comparison  Between  Exact  and  Interpolated  Values  of  p    and  u 

(m  =  10,  V  -  8,  X/v  -  *01) 


n  ■ 

8 

n  = 

16 

2 

s 

.0169 

.0256 

.0361 

.0484 

.0169 

.0256 

.0361 

.0484 

P  (exact) 

.2538 

.2654 

.2812 

.3029 

.3960 

.4285 

.4730 

.5300 

P^(lnterpolated) 

(.2540) 

(.2656) 

(.2815) 

(.3031) 

(.3966) 

(.429?) 

(.4735) 

(.5301) 

*2 

b)  (exact) 

.0718 

.0787 

.0886 

.1030 

.1674 

.1957 

.2373 

.2953 

*2y 

u)^  (Interpolated) 

(.0720) 

(.0789) 

(.0888) 

(.1031) 

(.1681) 

(.1966) 

(.2381) 

(.2955) 

*2       *2  *2 

0        =   U        -  p 

.0072 

.0083 

.0095 

.0113 

.0106 

.0121 

.0136 

.0144 

*2       *2  *2 
I  I 

(.0075) 

(.0084) 

(.0096) 

(.0112) 

(.0108) 

(.0124) 

(.0139) 

(.0145) 

It  may  be  noted  that  In  this  example,  the  monotone  functions  of  both  p 
*2  2 

and  u     on  s    are  slightly  postlvely  accelerated.    Consequently,  the  values 
obtained  from  linear  Interpolations  consistently  overestimate,  though 
nojUlglblyt  the  exact  values  as  demonstrated  In  the  above  table,  Hc\;ever, 
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*  *2  2 

the  4*li<jruc:turf Htic  of  the  monotone  functions  of  P    and  ui     on  d  varies 

g 

with  the  values  of  m>  n>  v  and  X'v  .    For  example^  given  m  =  10>  v  =  8> 

*  *2  2 

X/v  =  «0S»  and  n     16»  the  functions  of  p    and  ui     on  s    become  negatively 
accelerated*    Therefore^  whether  the  Interpolated  value  underestimates  or 
overestimates  the  exact  one  depends  on  other  parametric  values  (e«g*» 
m,  Tif  V  and  X/v)  being  considered*    In  general »  the  discrepancies  are 
very  small  when  linear  Interpolation  over  an  Interval  length  of  .01  Is 
applied  In  our  present  problem.    Approximations  of  p    and  w  for 
nontabulated  values  of  m»  n»  and  X/v  can  also  be  obtaj.ued  satisfactorily 
by  linear  Interpolations* 
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3*    A  Numerical  Example 

To  Illustrate  the  use  of  these  tables  in  actual  data,  the  example 

presented  in  Table  7  of  Lewis,  Wang,  and  Novick  (1973)  was  reanalyzed 

employing  these  tables*    There  were  35  children  taking  a  12-lten)  test* 

The  sufficient  sample  statistics  for  ou£  analysis  were  m  =  35,  n  =  12, 

g»  =  1*2287,  and       -  Kg    -  g*)^/m  =  *9175/35  =  *02621»    The  same  prior 
^  i 

distribution  for       (namely,  v  «  8,  X/v  =  .25/8  =  •03125)  was  adopted* 
For  notational  convenience,  we  shall  denote  p  (m,  a,  X/v,  s  )  and 
to    (m,  n,  X/v,  s  )  as  the  values  of  p    and  «     for  given  m,  n,  X/v  and 


.02621)  can  be  approximated  by  interpolating  between  these  two  values: 


.OA,  .03)  =  .5566,  we  have 


P*(30,  12,  .OA,  .02621)  =  .5A13  . 

Th2  next  step  is  to  interpolate  between  P*(30,  12,  .03,  .02621)  and 
* 

P  (30,  12,  .OA,  .02621)  to  obtain: 

p*(30,  12,  .03125,  .02621)  =  .5023  . 
Following  the  same  procedure,  P*(AO,  12,  .03125,  .02621)  was 


approximated  using  the  table  for  m  =  AO,  n  «  12: 


P*(AO,  12,  .03,  .02)  -  .AAA9 
P*(AO,  12,  .03,  .03)  =  .A970 


gives  p*(AO,  12,  .03,  .02621)  =  .A773; 


gives  p*(AO,  12,  .OA,  .02621)  =r  .5192; 


* 

thus»  p  (AO,  12,  .03125,  .02621)  =  .A825. 


8 


Finally,  we  interpolate  between  m  =  30  and  m  =  AO  to  approximate  the 
valite  for  m  *  35 : 


p  (35,  12,  .03125,  .02621)  ^  ^£(.5023  +  .4825)  =  .A92A 


This  value  is  very  close  to  the  exact  value  (.4920)  obtained  from  our 

*2 

previous  analysis.    In  the  same  way,  u    (35,  12,  .03125,  .02621)  can  be 
approxlnut:eci  from  available  tables.  First, 

u*^(30,  12,  .03,  .02)  =>  .2255  ■  ^- 

*j  \  gives  u    (30,  12,  .03,  .02621)  -  .2533; 

o    (30,  12,  .03,  .03)  "  .2702  ) 


and 


0)    (30,  12,  .OA,  .02)  -  .2718  ^)  ^- 

*j  \  gives  u    (30,  12,  .OA,  .02621)  -  .2988; 

u    (30,  12,  .OA,  .03)  =  .3153  \ 

so  that  u*^(30,  12,  .03125,  .02621)  -  .2590. 


Secondly, 

u*^(AO,  12,  .03,  .02)  »  .2028))  ^- 

^2  (gives  u    (AO,  12,  .03,  .02621)  =  .2336; 

u    (AO,  12,  .03,  .03)  =  .252AJ 

and 

u*^(AO,  12,  .OA,  .020  «  .2AA7^ 

> gives  u    (AO,  12,  .OA,  .02621)  -  .2932; 

u    (AO,  12,  .OA,  .03)  -  .2932  \ 

30  that  w*^(40,  12,  .03125,  .02621)  ^  .2388. 

Finally,  we  arrive  at 

*2 


(0    (35,  12,  .03125,  .02621)  ^  %(.2590  +  .2388)  =  .2489 


*  *2 

Thus,  the  approximate  values  of  p    and  a     for  the  present  data  have 


been  obtained: 


and 


p*  ^  .4924 


*2       *2  *2 
a     -  0)     "  p     m  ,0064  . 


29i 


9 


Now  applying  formulas  (1)  and  (2) >  the  posterior  marginal  mean  estimates 

(w.)  of  Y    (thus  TTj  of  TT.)  given  g  and  the  corresponding  posterior 
2 

variances  0^  (or  standard  deviations  o^)  can  easily  be  computed.  The 
results  obtained  from  the  present  analyses  are  compared  with  the  previous 
results  produced  by  the  program  HAKFRO  described  In  Lewls^  Wang»  Novlck^  1973. 
In  the  table  presented  below»  estimates  from  the  present  approximate  method 
are  given  together  with  those  exact  estimates  (enclosed  In  parentheses) 
obtained  from  MARPRO  output. 


Posterior  Marginal  Estimates  of  y^,    ^  >  and  0^ 


X,  -  8 

3 

X    «  9 

j 

X,  =  10 

J 

X,  =  11 

J 

X,  =  12 

J 

.7961 

.8304 

.8656 

.9036 

.9606 

''J 

(  .7961) 

(  .8305) 

(  .8657) 

(  .9036) 

(  .9606) 

1.0877 

1.1290 

1.1745 

1.2287 

1.3280 

(1.0878) 

(1.1291) 

(1.1746) 

(1.2287) 

(1.3279) 

.1033 

.1020 

.1011 

.1007 

.1020 

(  .1029) 

(  .1018) 

(  .1010) 

(  .1007) 

(  .1018) 

ERIC 


These  comparisons  clearly  show  that  there  are  practically  no  differences 
between  the  approxloate  and  exact  results.    Accordingly,  the  posterior 
probabilities  prob(Trj  ^^^qIs)  approximated  by  our  present  analysis  are 
not  expectedTtTHlTfer  significantly  from  the  exact  probabilities  In  our 
previous  analysis.    This  Is  so  because  the  normal  approximations  to  these 
probabilities  have  been  found  adequately  accurate.    The  posterior 
probabilities  for  '^^  *  .70(.05).90  computed  from  formulas  (9)  and  (10) 
using  the  current  approximate  estimates  of  p    and  0     are  presented  be]ow 
to  compare  with  the  exact  probabilities  (enclose<t  In  parentheses)  obtalivid 
by  numerical  Integrations  with  the  program  MARPRO: 
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It 

0 

prob 

.90 

.0592 
(.0557) 

.1197 
C.117A) 

.2306 
(.2298) 

.A201 
<.A193) 

.7807 
(.7808) 

.85 

.2042 
(.2035) 

.3327 
<.33A8) 

.5055 
(.5082) 

.7096 
(.7110) 

.9356 
(.9379) 

.80 

.A  255 
<.A30A) 

.5850 
(.5898) 

.  7A75 
(.7502) 

.886A 
(,8873) 

.98A8 
(.9861) 

.75 

.6525 
(.6588) 

.7887 
(.7917) 

.8960 
(.8966) 

.96A3 
<.96A2) 

.9970 
<.997A) 

.70 

.8249 
(.8276) 

.9117 
(.9115) 

.9651 
<.96A2) 

.9908 
(.9905) 

.9995 
<>.999) 

The  small  discrepancies  between  the  exact  and  approxltnate  probabilities 
In  the  above  table  will  not  have  effects  on  our  decision  making  in  practical 
applications. 

Sometimes,  credibility  Intervals  may  be  of  Interest  to  an  investigator. 
They  can  be  approximated  using  formulas  (11)  and  (12).    For  our  present 
example,  we  have  computed  the  approximate  posterior  95%  credibility 
Intervals  of  u,  for  each  observed  x, : 


95%  confidence  interval  of  it^ 

'  8 

(.5991 

.9233) 

"  9 

<.6A17 

.9A26) 

10 

(.6863 

.9613) 

11 

(.7361 

.9792) 

12 

(.8165 

.9982)  ^^'^ 

11 


The  reader  may  check  the  exact  posterior  probabilities  given  in  Table  7  of 
Lewis,  Wang,  and  Novick  (1973)  to  convince  himself  that  these 
approximate  intervals  are  sufficiently  close  to  the  exact  intervals  which  are 
very  difficult  to  obtain  directly  from  tht  actual  posterior  marginal  density 
functions  of  Yj  • 

In  conclusion,  it  is  felt  that  these  tables  will  prove  useful  in  analyses 
of  m-group  proportion  data  (with  equal  sample  size  n)  without  recourse  to 
the  .program  MARPRO. 
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1-  Introduction 

In  two  previous  papers  (Novick,  Lewis,  and  Jackson,  1973;  Lewis^ 
Wang,  and  Novick,  1973),  the  problem  o£  estimating  proportions  in  m 
groups  was  studied  with  a  Bayesian  Model  II  approach,  using  the 
arc  sine  variance  stabilizing  transformation.    It  was  shouii  that 
Bayesian  Model  II  estimates  were  preferable  to  the  conventional 
sample  estimates  especially  when  ijp  (the  variance  o£  the  transformed 
variable  7^)  is  small-    This  gain  can  be  equated  to  substantial  savings 
o£  sample  size  in  data  collection.    An  extension  o£  this  work  is  the 
problem  o£  estimating  proportions  in  two-way  tables.    For  example,  a 
set  o£  t  tests  may  be  given  to  each  of  m  persons-    We  are  interested  in 
estimating  the  level  of  functioning  of  each  person  on  each  test. 
By  level  of  functioning  on  a  test,  we  tnean  the  percentage  of  correct 
responses  that  the  person  would  make  to  a  test  composed  of  all  of  the 
items  which  might  have  been  selected  for  the  particular  test.  The 
model  considered  is  the  so-called  Model  II  or  random  effects  model 
because  the  persons  and  the  tests  are,  respectively,  considered  to  be 
rardom  samples  from  larger  populations  of  persons  and  tests.    As  in  a 
two-^way  analysis  of  variance  design,  one  can  assume  that  the  variations 
of  performance  are  due  to  row  effects  (persons),  column  effects  (tests), 
and  interaction  effects.    Thus,  each  of  these  effects  can  be  separately 
estimated  and  then  combined  to  provide  estimation  of  the  proportions. 
This  estimation  procedure  would  find  an  application  in  the  area  of 

The  research  reported  herein  was  performed  pursuant  to  Grant  No^ 
OEG-0-72-0711  with  the  Office  of  Educationi  U,  S.  Department  of  Health, 
Education,  and  Welfare,  Melvin  R,  Novick,  Principal  Investigator. 
Contractors  undertaking  such  projects  under  Government  sponsorship 
are  encouraged  to  express  freely  their  professional  judgment  in  the 
conduct  of  the  project.    Points  of  view  or  opinions  stated  do  not, 
therefore,  necessarily  represent  official  Office  of  Education  position 
or  policy. 
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individually  prescribed  instruction.    After  completing  a  unit  of 
instruction*  each  student  is  given  a  poattest  unit  which  consists  of  a 
set  of  tests  on  related  skills.    Estimates  of  the  level  of  functioning  of 
fiach  student  on  each  skill  can  be  obtained  to  help  make  decisions  on  each 
individual's  progress. 

The  Bayesian  Model  H  approach  was  proposed  by  Lindley  (1971). 
Theories  and  solutions  for  the  general  linear  model  have  later  been 
discussed  in  some  detail  by  Lindley  and  Smith  (1972).    The  present 
paper  proposes  to  apply  the  Bayesian  estimation  procedures  to  two- 
way  tables  of  proportions.    Essentially^  these  procedures  incorporate 
the  collateral  information  provided  by  the  other  persons  as  well  as  by 
the  other  tests  into  the  estimation  of  a  single  proportion.  Consequently^ 
some  advantages  are  expected  over  the  conventional  sample  proportion 
estimates. 


2.    Basic  Model 


The  observed  number  of  successes         for  individual  i  on  test 
j  is  mapped  into  g^,^  by  the  Freeman-Tukey  (1950)  transformation; 


g,^=i|^slny^   +    sury^^j    .  (2.1) 


ERIC 


where  n,  is  the  number  of  items  in  test  j  .  We  will  assume  that  the 
are  binomially  distributed  with  parameters  n^  and  atid  that  they 

are  jointly  independent  given  the  ir^^  .    Under  these  assumptions^  the 
g  ,  are  jointly  independent  and  to  a  satisfactory  approximation  are 
normally  distributed  with  mean        -  sin  and  variance 

^ij  "  ^ j  "  ^  2)"^»  provided  n^  >_  8.    The  objective  of  this  and 

3^0 


related  LranstormaLions  is  variance  stabilization.    Tor  further 
discussion  on  thib  points  us  well  as  on  Llie  adequcicy  of  tin  approxim^jtioi^iy, 
tlie  reader  is  referred  to  Novick,  Kewis>  and  Jackson*  (1973jj  and 
Lewis,  Wang,  and  rjo^^ick^  (1973), 

To  proceed  further  >  we  must  specify  a  distribution  for  r>  the 
matrix  of  cell  means  y.,  •    If  we  treat  the  persons  and  tests  as 
independent  random  samples  from  appropriate  populations^  then  we  may 
follow  the  standard  development  of  random  effects  models  given,  for 
example,  in  Scheffe  (1959,  pp.  238-242)*    This  development  requires 
only  that  the  persons  and  tests  be  sampled  independently  of  each  other 
and  that  the  distribution  of  F  be  multivariate  normal,  given  the 
necessary  means  and  dispersion  matrix.    It  is  then  possible  to  define 
0,  a,,  i^,,  and  6..  such  that 


and  such  that  {a^},  t3j}>  and  tS^jJ         independent  normal  with  zero 
means  and  variances  <}>^,  <}>^,  and         respectively,  conditional  on  these 
variances  and  independent  of  0  • 

The  definitions  are  given  in  terms  of  expectations  of  y^j  with 
respect  to  the  population  of  persons  and  the  population  of  tests. 
We  indicate  these  expectations  l>yLT  '"^^^  ^  i*  i^^^pe^tlvelyt 


1 


3 


(2*2) 


(2.3) 


^I  "^'j^^IJ^  ^  ^  * 


(2*4) 


(2.5) 


(2.6) 
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In  many  cases  where  persons  and  tests  have  not  been  randomly 
sampled,  it  may,  nonetheless,  be  possible  to  characterize  our  beliefs 
about  the  values  of       as  exchangeable  for  the  group  of  persons  being 
tested  and  for  any  other  group  selected  from  the  population  of  interest ► 
In  addition,  a  sljuilar  statement  may  hold  for       and  the  population  of 
tests.    Finally,  our  beliefs  about  the  interaction  terms  (really 
residuals  from  a  simple  additive  model)  6^^  may  be  exchangeable,  at 
least  in  the  sense  that  W3  have  no  good  reason  to  expect  any  particular 
pattern  of  deviations  from  additivity  in  y^^  . 

Lindley  and  Smith  (1972),  among  others,  have  applied  the  work  oC 
De  Finetti  (1937)  and  Hewitt  and  Savage  (1955)  on  exchangeability  to 
situations  such  as  this.    If  we  are  willing  to  express  our  beliefs 
about  Y^j  as  described  in  the  previous  paragraph,  we  may  conclude  that 
{a^l  have  the  structure  of  identically  and  independently  distributed 
random  variables  conditional  on  some  parameter(s) .    Similar  statements 
hold  for  tBj}  and  (^^j)^ 

For  mathematical  convenience,  we  introduce  the  additional  assumption 
that  all  the  above-mentioned  conditional  distributions  are  normal ►  It 
immediately  follows  from  definitions  (2.4),  (2.5),  and  (2.6)  that  the 
expectations  of  a^,  P^,  and  6^^  are  zero^    Hence,  we  may  write 

•^(^l  V  "  C^^  ^"^^^  *a^^^l^'  ^^^^^ 
b(A|4^^)  «  f^"^^^^  exp(-  h  h^^^^lp  '  ^^'^^ 


and 
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A  final  assumption  required  at  this  stage  is  that  a,       and  A  are 
jointly  independent,  given  <t>^,         and       .    This  assumption  will  be 


reasionable  as  regards  a  and     whenever  the  choice  of  persons  is  unrelated 
to  the  choice  of  tests.    The  remainder  of  the  assumption,  namely  the 
independence  of  A  from  ot  and  &  jointly,  is  less  immediately  intuitive 
but  may  be  considered  reasonable  by  noting  that  it  is  equivalent  to 
the  assertion  that  knowledge  of  a  and  6  tells  us  nothing  about  the 
distribution  of  A,  which  may  be  a  justifiable  assertion  on  the  basis 
of  ignorance. 

We  have  now  reduced  the  problem  of  specifying  a  distribution  for 

r  (either  by  standard  methods  or  with  suitable  exchangeability  and 

independence  assumptions)  to  that  of  specifying  6  ,      ,         and  9^  »  In 

most  cases^  it  will  not  be  reasonable  to  assume  that  the  values  of 

these  parameters  are  known.    Consequently,  ve  suggest  the  following 

distributional  and  independence  assumptions:    take  <}>^,         and  to 

2 

be  independently  distributed  as  inverse  x    variables  and  denote  the 
degrees  of  freedom  and  sum  of  squares  parameters  for  these  distributions 
by  (v  ,  X  ),  (v.,  X^),  and  (v.,  X.),  respectively  (see  Novick  and 
Jackson,  1974,  Section  7-3).    Finally,  treat  6  as  locally  uniform  in 
the  range  of  interest  and  jointly  independent  of  <j>  ,  <J  a,  6,  and 

A  .    We  believe  these  distributions  will  satisfactorily  characterize 
whatever  vague  knowledge  we  may  have  about  the  overall  maan  of  y^^ 
and  its  component  variances.    For  reasons  discussed  by  Novick  (1969) 
and  by  Novick,  Lewis,  and  Jackson  (1973),  a  uniform  distribution  for 
6  will  be  acceptable;  however^  a  proper  prior  will  be  required  for  the 

variances  (<i>^,       and  <t>^).    Still  outstanding  is  the  issue  of  supplying 

2 

values  for  the  three  pairs  of  inverse  x  parameters^  We  deftr  discussion 
on  this  point  until  Section  4. 
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With  the  above  definitions  and  assumptions,  we  find  the  likelihood 
function  and  the  joint  posterior  distribution  of  t),  a,  P,  A,  41^,  and 
<t»g  given  G  =  (gy)  to  be: 

1  =  1,  2,  . . m;  j  =  1,  2,  . . . ,  t; 

and 

b(e,  a,       6,  ^glG) 

-%(in  +      +  2)  2 

"  exp{-  +  ^ap/*„> 

-%(t  +  V.  +  2)  2 
.frp  ^  exp{-  iiCXp  + 

-*£(rot  +  V.  +  2)  - 

*  exp{-  hCXj  +  ri:6^j)/<!.^)  ,  (2.10) 


respectively.    We  may  use  equation  (2.2)  to  include  V  explicitly  in  the 
joint  posterior  distribution  (2.10).    Specifically,  we  substitute 
Y^j  -  6  -       -  ^ij  and  leave  the  other  parameters  unchanged.  Sinc^ 

the  Jacobian  of  this  transformation  is  unity,  no  further  adjustments  to 
(2.10)  are  necessary.    Thus,  we  have 
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a'  ^g'  ^5' 

«  exp{-     tZig^.  -  Y.j)  /v.}  •  exp(-  Sj^^  +  £ap/*^) 

-h<t  +  V   +  2) 


-'sdnt  +  V   +  2) 
*5  exp{-         +  ^S6^j)/<^5)    .  (2.11) 


3.    Posterior  Joint  Modal  Estimates 

3.1    Joint  Modal  Estimates  for  the  Basic  Model 

Integrating  b(r,  6,  a,  6,  i>^\G)  in  equation  (2.11)  with 

respect  to  the  nuisance  parameters  <fi^,  and  we  obtain  the  posterior 
joint  distribution  of  r,  6,  a,  and 


b(r,  e,  a,  &\G) 

«  exp{-  h  22(gij  -  Yy)^/v^>    •   C\  + 


-him  +  v^) 


(3.1) 


For  the  posterior  joint  distribution  of  V  alone,  we  need  to  integrate 
expression  (3.1)  with  respect  to  6,  a,  and  P  from  equation  (3.1).  Explicit 
expressions  for  these  integrations  do  not  appear  to  us  to  be  possible. 
Therefore,  Vfe  obtain  the  joint  mode  of  T,  6,  a  and  P  as  estimates  of 
the  corresponding  vector  elements.    Differentiating  f  «  in  b(r,  6,  a,  i\G) 
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witli  ruspccl  to       a^>  &y  and  y^^  and  setting  the  deriv^ttives  to  z^^xo, 
a  system  of  equations  is  derived  to  be  solved  for  the  joint  posterior 
mode  of  Oj  a>       and  V: 


v>,j  -  -  i-^j  -  5  -       -  Bj)-  0  .  (3.2a) 


3t     .  -1,.       r,  % 

^      :  <C  +  t$-^a^  -  t^-^Y^.  -  6  -  e.)  =  0  ,  <3.2c) 


and 


+  nKf^        -  m^^  (Y.j  -  6  -  a.)  =  0 


<3.2d) 


where 


and 


=  <X^  +  Sa^)/(ra  +  v^)  ,  <3.3a) 
-  <Xp  +  Sej)/<t  +        ,  (3.3b) 


=  [X^  +  SS(Yj^j  _     _       _  e^)^]/(iQt  +  Vg)  .  <3.3c) 


Thus,  we  find  the  posterior  joint  modes: 

*  - 

"   '^a^^i*  "  ^"^  *  (3.4b) 
^4  "   *<'p(Y-j  -  Y..)  -   <g.    -  Y..)  ,  <3.4c) 

<f»g  +  t  +  Vj) 
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and 

0  =  Y.'  »  (3.4d) 

where 

W      =   4      (iJ.  = 


and  the  dot  notation  indicates  average  over  the  appropriate  i^idex*  Also, 
note  that  S<  -       =  0  • 

Tlie  joint  modal  estimates  o£  r>  6,  a,  and  3  then  can  be  obtained  by 
an  iteration  procedure*    The  usual  least  squares  estimates  6  =  g<*, 

^i     ^i'  *  "  ^'j  "  ^ij      ^ij  ^®  initial  values 

o£  0,  a^,        and  Y^^  •    Given  these  initial  values,  and  §^  are 

computed  from  (3*3)  and  used  to  obtain  improved  values  of       a,  S>  and 
^  via  (3.2).    Substituting  these  new  values  in  (3.3)»  the  foregoing  process 
is  repeated  to  refine  the  estimates  of       a,       and  §  .    This  iterative 
procedure  continues  until  some  convergence  criterion  is  reached*    It  should 
be  noted  that  it  may  converge  to  some  local  mode  i£  bimodaliny  or 
multimodality  exists. 

Looking  at  the  expressions  in  (3.4),  we  find  Lhat  these 
Bayesian  modal  estimates        are  weighted  averages  of  observed  g^^  and 
the  sum  8  +      +       .    In  terms  of  (3.4b"d)>  it  is  seen  that  and 
$j  are  functions  of  row  averages  y^. >  column  averages  y-^  and  the  overall 
average  y..  .    Under  the  basic  model  described  In  Section  2,  the  cell 
mean  y^^  for  a  specific  cell  (i^  j)  was  assumed  to  be  normally  distributed 


with  mean  0  +  a.  +  S,  and  variance  ({>j.*  conditional  on  6.  a..  6,*  and  <f>^  . 
1       j  5'  '    i'  j' 

h 

Thus,  the  weight  — r  assigned  to  the  observed  g.j  is  a  Bayesian 

9^  -r  12 
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reliability  estimate  of  g^,^  given  fixed  i  and       .    In  obtaining  estimates  of 

y^y  observed  g^^  are  regressed  towards  the  value  S  +  Ct^  +       .    We  may 

also  remark  that  with  the  definition  of         given  by  (2.2),  y^.  -  y.. 

and  y,^  -  y,.  would  b^  .ease  squares  estimates  of       and  6^,  respectively, 

it  ^ 

provided  y  ,  were  observable.    In  this  case,  w    and       can  be  regarded  as  tueir 
corresponding  reliability  estimates.    Accordingly,  the  estimates  for 
are  seen  to  be  Yj ►  "         regressed  towards  their  common  value  5.  "  0  . 
This  interpretation  extends  to  the  case  of  . 

Having  obtained  the  posterior  estimates  of  y^y  estimates  of  ii^^  can 
be  approximated  by; 

following  the  previous  study  of  the  one-way  case  (Novick,  Lewis,  and 
Jackson,  1973,  p.  24).    In  obtaining  estimates         of  proportions,  the 
regression  of  sample  proportions  p^^  towards  some  common  value  corresponds 
to  that  of  g^j  since        are  monotonlc  increasing  transformations  of  ^ 

3.2    A  Special  Case  Where  All  lests  Are  of  Equ.n1  Length 

In  some  applications,  the  set  of  related  tests  may  have  the  same  number 
of  items,  i.e.,  n_j  =  n  .    This  means  that  the  error  variances  within  each 
cell  are  all  equal  (v^  *  v  =»  ^-^--^-j).    In  this  case,  the  solutions  for 
(3.2)  can  be  simplified.    The  joint  modal  estimates  of  T,  e,       and  S 
given  G,  can  now  be  written  as; 

^Ij  "       ^ij  ^       '  ^6^5  +  6^  +  l^) 

=       g^^  +  (1  -  Wj)[g..  +  -  g..)  +  ^g(8^j  "  8^0]         (J. 6a) 

\  "  ^a^^±*  "  (3.6b) 
Bj  »  "  8-0  (3.6c) 
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and 


where 


9  =  g..  =  llg    /mt  (3.6d) 


w-.  =  T    <  w   :   <  w  ^ 


(3.7) 

and  ^  ,  ^>-,  |.  are  estimated  from  (3.3)  , 

ot      p  0 

Equation  (3.6)  indicates  that  the  Bayesian  Model  II  joint  estimates 
of  the  transformed  proportions         can  be  written  explicitly  as  linear 
combinations  of  observed  values  g^^^  deviation  row  means  g^^  "  g**> 
deviation  column  means  g.^  -  6**j  and  the  overall  mean  g^^  .    The  weights 
^a*  ^S*         ^6  interpreted  as  reliability  estimates  of  the 

components  g  .  -  g..>  g.,  ^  g. . ,  and  g_  .  Consider  6>  <x, ,  and  given, 
the  basic  assumptions  in  Section  2  imply 

Var(g^^|e,  a^,  B^)  -       +  v 

and 

Var(Y,^l0,  a^,  B^)  -  . 

Hence,       is  a  reliability  estimate  of  g^y  conditional  on  (j,  a^^,  and  , 

Thus,  it  is  seen  that  joint  estimates  o£  y,,  are  observed  g,,  regressed 

towards  6     a.  +  L  . 

^  3 

The  reliability  interpretations  of  w    and  w   may  be  less 

ot  p 

straightforward.    However,  borrowing  from  the  results  of  classical 
random  effect  ANOVA,  we  obtain 

Var(g^.  -  g..)  «  (^^  ^    ^  _    '    *a     ^'^^^^  ' 

2 

since  i:(g^.  -  g..)  /(m  -  1)  is  an  unbiased  estimate  of  Var(g^.  -  g..)  and 
i  f\ 

ti:(g^.  -  g. .)  /(m  -  1)  "  t<}>^  +       +  V  from  the  expected  mean  squares 
in  random  effect  ANOVA.    The  sample  statistic  g^.  "  g..  is  an  estimate 
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of  the  row  effect        whose  variance  is  assumeU  to  be       *    Thus,  u^^ 
can  be  regarded  as  an  estimate  of  the  reliability  of       =  g^,  -  , 
Since  5*  =  0,  we  can  write  the  joint  modal  estimates  a,  of  a.  in  (3.6)  as 

^i  *  *^a^^i*  -  g'O  +  (1  -  w^^)6t.  =  w^a^  +  (1  -  0)^)5.  . 

It  is  clear  then  that  the  et^,  being  a  weighted  average  of  the  least  squares 

estiT:.ator  &^  and  the  canmon  value  5.,  are  regressed  towards  a.  =  0  . 
i 

The  same  interpretation  also  extends  to  the  case  of  the  joint  modal 
estimates       of  =  t^^l^^  +  (1  ^  Wg)&.3  . 

Returning  to  (3*6),  we  can  write 

^ij  "  ^ij         "  ''S^^S..  +  ^Cg^.  -  g.O  +  t^JgCg-j  -  g")  -  g^jl  • 

(3.8) 

Hence,  the  regrtssioD  of  g . ,  towards  the  estimate  9  +  a  ,  +    ,  = 

g*  •  +  *^a^^i'  *  ^"^  ^  ^6^^'j  "  depends  on  the  particular  row  i  and 

column  j  .    For  instance,  if  the  observed  g^^  is  greater  than  the  value 

6+       +  &y         will  be  smaller  than  g^^  .    The  relative  roles  of  a  specific 

row  i  and  column  j  in  determining  the  direction  of  the  regrtission  of  g^^ 

vii  tliti  reliability  estiutatec  (j^  and  .  Tor  exauple^  i£  w  is  much 
larger  than  g^^  will  be  regressed  mostly  towards  a  combination  of  g^. 
and  g..  . 

In  passing,  it  is  also  interesting  to  note  that  y  .  «  ^"^a*^^ 

j 

is  a  weighted  average  of  g^.  and  g.*: 

Y^.  -  [1  "  (1  -  t^^)a  -  +  (1  -  t^^)a  -  u)^)g.. 

where  _ 

a  0 


ERIC 


(I).  =  1  -  ci  -  (I).)  a  -  (I) )  ^T-^ 

3u0 


I } 


is  Interpreted  as  an  estimate  of  the  reliability  of  g^^.  conditional 

.-1 


on 


e  and  6  since  Var(e,.i6»  6)  =       +  t~\^.  +  v)  and  Var(v..|e»  6)  = 
+  t     <j^g  under  the  assumptions  discussed  in  Section  2,  Therefore, 
estimates  y^,  of  y^,  are  obseived  row  averages  g^-  regressecl  to  the  overall 
average  g.,  ,  Similarly, 


where 

to^„  =  1  -  (1  -  w^)(l  -  w„)  =  ^ 


(3.10) 


is  an  estimate  of  the  r-^liability  of  g*^  conditional  on  6  and  a  . 

3*3    A  Generalized  Case  of  the  Present  Model 

Although  we  have  discussed  the  problem  of  estimating  proportions  in 
two-way  tables  in  the  context  of  testing,  this  same  model  can  be  extended 
to  a  more  general  case  where  the  indices  n^^  of  thft  binomial  distributions 
for  x^^  in  cells  (i,  j)  are  all  unequal.    For  example^  one  may  be 
interested  in  simultaneously  estimating  proportions  of  female  students 
in  t  different  majors  (Science,  Art,  etc*,)  for  each  of  m  state 
universities.    In  this  case,  we  may  take  samples  of  different  sizes  n^^  for  each 
combination  of  majors  and  universities.    Replacing  all  v,  by  v.*  =  ^ 


j        ^ij      4n^^  +  2 
in  (3.1),  we  obtain  the  posterior  joint  distributions  of  T,  6,  a,  and 

0  for  this  general  case.    Thus,  the  joint  modal  estimates  of  T,  6,  a, 

and  6  can  be  found  by  solving  system  (3,2)  iteratively  except  substituting 

for  V*  in  (3,2a),     The  estimates  tt,*  of  u,,  are  also  obtained  f"m 
ij  3  ^3 

(3.5)  with  n^^  substituting  for  n^  , 
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It  may  be  noted  that  in  this  generalized  application ,  our  problem 
is  similar  to  that  treated  by  Lindley  (1972)  with  two  differences. 
Firstly,  Lindley  studies  the  general  two-way  ANOVA  design  so  that  there 
usually  are  replicated  observations  within  each  cell.    In  the  present 
case,  there  is  only  one  proportion  observed  for  each  cell.  Secondly, 
we  have  a  simpler  case  where  all  within  (error)  variances  C^^j) 
known,  while  Lindley  deals  mostly  with  unknown  within  variances. 


In  Section  2»  ^  ,  f^^,  and       are  apriori  assumed  to  have  independent 
ot     p  o 

iaverse  chi-square  distributions  with  parameters  (v^,  X^) ,  (v^,  X^j) , 
and  (Vg,  Xg),  respectively.    In  practice,  the  investigator  must  supply 
values  for  these  three  pairs  of  parameters  to  make  the  analysis  feasible. 
It  has  been  argued  by  Novick,  Lewis  and  Jackson,  (1973),  chat 
in  the  absence  of  any  specific  information,  a  reasonable  choice  fou 
the  degrees  of  freedom  parameter  of  the  inverse  chi-square  distribution 
is  8  .    If  we  accept  this  choice,  the  problem  is  reduced  to  chat  of 
specifying  X^,  X . ,  and  X.  . 

CX       p  o 

According  to  tho  assumptions  made  in  Section  2»  the  prior 

marginal  distribution  of  y^^  conditional  on  6,  <{>^,         and       is  normal 
with 


Further  Discussion  of  the  Prior  Distributions 


and 
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Now  If  wc  ask  the  investigator  about  what  the  variance  of        for  a 
randomly  chosen  person-test  combination  (i>  j),  we  obtain  an  estimate 
of  the  sum  *p     ^5  ^    This  variance  would  usually  be  small,  for 

example,  between  *04  and  .02.  We  can  further  equate  this  estimate  to 
the  expected  value  of       +  ^5  *    Since  independent  inverse  chi- 

square  distributions  are  assumed  for        4*^,  and        we  obtain 

-  X  X  \ 

ct  6  6 


Wow 


»  for  illustrative  purposes,  assume  the  estimate  of       +  4>g 


+       is  .02,  i.e. , 


X  X.  X. 

+  +  o  "  ^02  .  (4.3) 


v-2         V.  "2      V.  -  2 
a  $0 

Combining  (4.3)  with  the  choice  ^^j^  *  ^5  *  ^*       should  take 

V   +      +  V.  =  .12  .  (4.4) 

ct       6  o 

The  Investigator  can  now  divide  the  total  given  in  (4.4)  among  the  three 
sum  of  squares  parameters  according  to  his  prior  beliefs  as  to  the  relative 
importance  of  person,  test  and  person  by  test  interaction  effects  on  the 
transformed  Y^j  *    However,  he  should  not  set  any  of  these  parameters 
equal  to  zero  for  reasons  discussed  in  Novick  (1969)*    Thus,  in  the 
absence  of  specific  Information,  he  might  choose 

=  =  .04  *  (4*5) 

In  the  next  section,  we  shall  examine,  among  other  ithings,  the  effect  of 
these  choices  on  the  estimates  of  . 
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5.    Numerical  Examples 

To  illustrate  the  application  of  tlie  present  model,  a  set  of  data  is 
constructed.    There  are  25  persons  taking  5  related  tests,  each  having  8 
items,    Tha  observed  proportions  of  correct  answers  are  given  in  Table  1. 
These  data  were  analyzed  with  different  prior  inverse  chi-square  distri- 
butions for  <{>^,  <{>g,  and       .    In  Table  2,  estimated  proportions  U^^j)  based 
on  prior  specifications  v    =       =       =  8  and  X    -       =  .028  are 

Ol  p  O  Ot  p  O 

presented.    It  may  be  noted  that  since  u)^(«  .0071)  is  negligible  compared 
with  u)^  and  t^y^  ^  the  estimates  y^^  are  nearly  completely  regressed 
towards  the  combination  w^(g^.  -  g..)  +  ^g^S'j  "  =  6  +  + 

(see  equation  3*6a).    Thus,  the  individually  observed  g^^  plays  very  little 
role  in  estimating        except  through  its  contribution  to  g^,,  g*^, 
and  g*.  .    Accordingly,  the  estimates        are  largely  dependent  on  the 
combined  row,  column,  and  overall  averages  of  observed  proportions. 

In  order  to  study  the  effects  of  prior  parameters  (v^,  ^^),  (v^j,  A^)> 
and  (v^,         on  the  estimates  ^^^f  these  data  Were  also  analyzed  with 

^  "  ^  =  ^6  =  ^>  ^Q-^^-  6. 

X    =       =       =  ,10.    The  results  were  presented  in  Tables  3  and 

a       6  0 

respectively.    As  can  be  seen  from  these  tables,  Bayesian  estimates  oC 

*/3*         ^6         larger  for  bigger  prior  estimates  ^^^^6* 
A^/v^)  of  these  variances.    However,  the  increment  of       is  smaller  since 
its  estimate  is  dominated  by  the  sample  information  (with  weight  mt 
versus  v^).    Consequently,  as  prior  estimates  of  these  variance  components 
increase,  w    and       become  comparatively  larger  while  Wj.  does  not 
change  much.    In  general,  there  Were  not  substantial  differences  among 

estimates  fr,  ^  given  in  Tables  2,  3,  and  4*    They  reveal  the  common 
ij 
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trend  of  regressing  y^^  towards  6  H      +       and  assigning  very  little 

weight  to  specific  transformed  cell  proportions  g^^  .    As  tu^  and  (Ug 

increase^  estimates  It,,  will  increase  for  those  (i^  j)  cells  whose 

marginal  averages  g^.,  g-_,  are  both  bigger  than  the  overall  average 

g..  -    Conversely,  if  g^.  and  g-^  are  both  smallrr  than  g-.,  corresponding 

estimates  it*,  will  be  lower  for  higher       and  o)^  . 

ij  ^        a  6 

Finally,  we  ^ay  remark  that  for  this  data  set,  the  classical  estimate 

2 

of       is  negative.    The  sample  statistic    ^^(S^j  "  S^*  "  S'j  +  g")  , 

vhose  tfxpected  value  provides  an  estimate  of       +  v,  is  2.1491  ,  Therefore, 

the  classical  estimate  of       based  Dn  expected  mean  squares  is  found  to  be 

$^  -  U^CgjLj  "         '        ^  g">^l/(™  -  ^^^^      1)  -  V  -  -.0070  -  Tor 
reference,  we  also  calculated  classical  estimates  of  <j>^  and       based  on 
-  8'*)^      -6878  and  Z(g,j  -  g..)^  -  .00495: 

*a  "  t^*"  ^^^i*  ^  S")^/Cm  -  1)  -       -  v]  =  .0242 

and 

5-g  =  ^tm  i:(g.^  -  g..)^/(t  -  1)  -  1^  -  V]  -  .00034  . 

It  is  suspected  Lhat  in  the  present  context ^  the  classical  estimate  of 
(f^  +  V  is  based  on  only  one  observation  per  cell  so  that  it  is  subject 
to  large  variations  and  thus  highly  un^.table. 
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Table  1:    Observed  Proportions 


m  ^  25 

t  s  5 

rt  =3  8 

Subject/Test 

1 

2 

3 

4 

5 

Average 

1 

.875 

.750 

1.000 

.750 

.875 

.850 

1.149 

2 

.750 

.625 

.750 

.500 

.875 

.700 

.975 

3 

.875 

1.000 

1.000 

.875 

.875 

.925 

1.254 

4 

.750 

.500 

.625 

.750 

.750 

.675 

.947 

5 

.750 

.875 

.625 

.750 

1.000 

.800 

1.098 

6 

.875 

.625 

.750 

.500 

.625 

.675 

.951 

7 

1.000 

.875 

.875 

1.000 

1.000 

.950 

1.303 

8 

.875 

.875 

.750 

.875 

.625 

.800 

1.076 

9 

.750 

.875 

.750 

1.000 

.875 

.850 

1.149 

10 

.875 

1.000 

.625 

1.000 

.750 

.850 

1.175 

11 

.875 

.875 

.750 

1.000 

.875 

.875 

1.177 

12 

1.000 

1.000 

.875 

.875 

1.000 

.950 

1.303 

13 

i.eoo 

1,000 

1.000 

1.000 

1.000 

1.000 

1.401 

14 

1.000 

1.000 

.750 

.875 

1.000 

.925 

1.275 

15 

.750 

1.000 

.875 

.625 

.875 

.825 

1.126 

16 

.750 

.875 

.625 

.750 

,625 

.725 

.997 

17 

.500 

.875 

.625 

.750 

.625 

.675 

.951 

18 

.875 

.375 

.500 

.625 

,500 

.575 

.859 

19 

.500 

.375 

.375 

.625 

.750 

.525 

.80? 

20 

.625 

.500 

.625 

.500 

.250 

.500 

.734 

21 

.750 

1.000 

.750 

1.000 

It  000 

.900 

1.248 

22 

.875 

.875 

1.000 

.875 

.750 

.875 

1.177 

23 

.750 

.625 

.625 

.750 

.625 

.675 

.946 

24 

.750 

.875 

.750 

.500 

.875 

.750 

1.026 

25 

1.000 

1.000 

.750 

1,000 

1.000 

.950 

1.324 

Average 

.815 

.810 

.7'+ 5 

.790 

.800 

.792 

1*113 

1.129 

1,039 

1.101 

1*112 

g..  = 

1.099 
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Table  2: 

Estimates 

of  Proportions 

V     =  V     =  V. 

=  8,    X  = 

a 

X,  =  X,  =  .028 

Subject/Test 

1 

2 

3 

4 

5 

1 

.843 

.850 

.803 

.835 

.842 

2 

.762 

.770 

.714 

.752 

.761 

:i 

.886 

.894 

.849 

.879 

.884 

U 

.748 

.755 

.699 

.739 

.746 

5 

.820 

.829 

.775 

.812 

.820 

b 

.751 

.758 

.702 

.740 

.747 

7 

.906 

.911 

.868 

.899 

.904 

8 

.811 

.819 

.765 

.803 

.807 

9 

.842 

.851 

.800 

.837 

.842 

10 

.854 

.863 

.811 

.848 

.852 

11 

.855 

.862 

.813 

.849 

.853 

12 

.906 

.912 

.868 

.898 

.904 

13 

.939 

.945 

.907 

.934 

.938 

14 

.896 

.902 

.856 

.888 

.894 

15 

.832 

.842 

.790 

.824 

.831 

16 

.773 

.782 

.725 

.764 

.770 

17 

.748 

.760 

.701 

.741 

.747 

18 

.704 

.710 

.651 

.693 

.699 

19 

.675 

.684 

.623 

.666 

.674 

20 

.662 

.671 

.611 

.652 

.658 

21 

.883 

.892 

.844 

.878 

.883 

Li. 

o  c  c 

.obi. 

.040 

23 

.747 

.755 

.698 

.739 

.744 

24 

.787 

.796 

.740 

.777 

.786 

25 

.914 

.920 

.876 

.907 

.912 

'^^  -  .0068,       =  .0023,       =  .0002 


u)    =  .5357,  w„  =  .6620,  Uj^  =  .0071 
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Table  3:    Estimates  of  Proportions 


V 

a       S  5 

=  8,    X  = 
*  ct 

6  5 

.06 

Subject/Test 

1 

2 

3 

4 

5 

1 

.849 

.856 

.801 

.838 

.847 

2 

.753 

.762 

.695 

.739 

.753 

3 

.897 

.908 

.855 

.889 

.895 

4 

.737 

.745 

.677 

.727 

.735 

5 

.821 

.832 

.766 

.811 

.824 

6 

.741 

.748 

.66  J 

.725 

.735 

7 

.920 

.925 

.875 

.913 

.919 

8 

.811 

.821 

.756 

.802 

.806 

9 

.847 

.858 

.796 

.843 

.847 

10 

.861 

.873 

.808 

.855 

.857 

11 

.862 

.871 

.811 

.856 

.860 

12 

.920 

.928 

.875 

.910 

.919 

13 

.956  : 

.962 

.920 

.950 

.955 

14 

.909 

.917 

.861 

.899 

.907 

15 

.835 

.849 

.785 

.824 

.835 

16 

.766 

.778 

.707 

.756 

.762 

17 

.736 

.752 

.679 

.729 

.735 

18 

.687 

.691 

.620 

.672 

.678 

19 

.650 

.660 

.586 

.640 

.651 

on 

*  u££ 

21 

.893 

.905 

.847 

.889 

.895 

22 

.862 

.871 

.816 

.853 

.859 

7i5 

24 

.782 

.794 

.726 

.769 

.782 

25 

.929 

.936 

.884 

.922 

.927 

^    ■  .0100, 

$g  -  .0049, 

$g  =  .0005 

(u    =  .6256, 

(u-  =  .8027, 

(Ug  =  .0153 

3J8 
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Table  A: 

I^stlmates  of 

Proportions 

CI  e 

0       *  a 

A^.X^=  .10 

ect/Test 

2 

3 

4 

5 

1 

.853 

.860 

.803 

.8A0 

.851 

2 

,746 

.755 

.682 

.729 

.7A7 

3 

.913 

.862 

.897 

.90A 

A 

.728 

.734 

.660 

.716 

.725 

5 

.821 

.834 

.760 

.811 

.827 

6 

.734 

.739 

.666 

.713 

.725 

7 

.932 

.935 

.883 

.92A 

.930 

8 

.812 

.822 

.7A9 

.80] 

.803 

9 

.850 

.863 

.795 

.8A8 

.051 

10 

.866 

.880 

.807 

.862 

.861 

11 

.867 

.877 

.811 

.863 

.865 

12 

.932 

.939 

.883 

.920 

.930 

n 

.969 

.975 

.932 

.963 

.968 

lA 

.920 

.928 

.866 

.307 

.918 

15 

.837 

.855 

.783 

.82A 

.837 

16 

.760 

.775 

.69A 

.7A9 

.755 

17 

.725 

.745 

.663 

.719 

.725 

18 

.672 

.672 

.595 

.65A 

.660 

19 

.628 

.638 

.556 

.618 

.631 

20 

.613 

.623 

.5Ai 

.593 

.601 

21 

.900 

.915 

.851 

.899 

.905 

22 

.867 

.877 

.819 

.858 

.863 

23 

,727 

.736 

.659 

.715 

.721 

U 

.778 

.793 

.717 

.762 

.779 

15 

,940 

.948 

.891 

.933 

.939 

=  .OlAl,       =  .0095,       ■=  .0008 
(J   =  .7001,  w„  =  .8866,  and       =  .0257 
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Marginal  Distributions  for  the  Estimation  of  Proportions  in  Two-Way  Tables 

by 

Mlng-mei  Watig 
Thft  Americau  College  Testing  Program 
and 

Charles  Levis 
University  of  Illinois 

1.  Introduction 

The  Bayesian  Modr    II  technique  for  simultaneous  estimation  of 
proportions  in  m  groups  has  recently  been  extended  to  provide  estimates 
of  proportions  in  two-way  tables  by  Wang  and  Lewis  (1973) t    The  random 
effects  analysis  of  variance  technique  is  applied  to  the  transformed 
(observed)  proportions 

where  3;^,^  is  the  observed  number  of  correct  answers  for  person  i  on  test  j 
of  n^  items  (Freeman  and  Tukey,  1950),    The  g^^  are  then  assumed  to 
be  approximately  normally  distributed  with  mean  '^^^^^  sin'^^/r^j ,  where  the 
TT^^  are  the  trvie  proportions  of  successes  for  person  i  on  test  j)  and  known 
variance  v*  *  (^n*  +  2)  ^  .    The  next  step  is  to  express  y,  *  as  a  sum  of  the 


The  research  reported  herein  was  performed  pursuant  to  Grant  No, 
OEG-0-72-0711  with  the  Office  of  Education,  U,  S.  Department  of  Health, 
Education,  and  Welfare,  Melvin  R.  Novick,  Principal  Investigator. 
Contractors  undertaking  such  projects  under  Government  sponsorship  are 
encouraged  to  express  freely  their  professional  judgment  in  the  conduct 
of  the  project.    Points  of  view  or  opinions  stated  do  not,  therefore, 
necessarily  represent  official  Office  of  Education  position  or  policy. 
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overall  effect  9,  person  effect  a^,  test  effect         and  person  by  test 
interaction  effect  5^^  (these  effects  are  defined  in  terms  of  expectations 
of  Y^j  with  respect  to  the  appropriate  populations  of  persons  and  tests,  see 
Wang  and  Lewis,  1973,  Section  2).    Applying  the  exchangeability  theorem 
(De  Vinetti,  1937),  we  obtain  estimates  of  y^^j  by  the  Bayesian  Model  II 
procedure  which  incorporates  not  only  the  information  provided  by  the 
performance  of  all  persons  on  the  test  j,  but  also  the  information  contained 
in  the  performance  of  the  person  i  on  all  other  tests  into  estimation  of  a 
single         ,    The  resulting  Bayesian  estimates  exhibit  a  regression  of  the 
least  squares  estimates  *  ~  8*j  "  and 

\j  "  ^ij  "  ^i*  "        ^  ^"  towards  their  respective  averages  fi.,  S.,  and 
6..,  which,  in  the  linear  model,  are  each  zero.    Because  we  are  making 
use  of  collateral  information  (provided  by  other  tests  as  well  as  other 
persons)  in  estimating  a  specific  proportion  ir^^,  it  is  expected,  as  in 
the  one  way  m-group  proportion  case  (Novick,  Lewis,  and  Jackson,  1973), 
that  some  advantage  will  be  gained  over  conventional  methods. 

The  earlier  paper  (Wang  and  Lewis,  1973)  provides  us  with  the  joint 
modal  estimates  of  a^,  B^,  and  9  .    However,  in  applications  to 

decision  making  in  the  context  of  individually  prescribed  instruction 
(IPI),  the  posterior  marginal  distributions  of  the        will  be  more  useful. 
What  Is  required  in  such  applications  is,  for  each  test,  the  aposteriori 
probability  tliat  a  person  ^s  test  score  is  larger  than  some  prespecified 
level,  and  this  is  required  for  each  person  and  each  test.    As  in  the 
case  of  estimating  proportions  in  m  groups,  algebraic  expressions  in  closed 
form  for  these  marginal  distributions  do  not  seem  to  exist.    Hence,  the 
marginal  distributions  of  the  Y^^j  will  be  studied  numerically  in  the  present 
paper.    In  particular,  we  shall  attempt  to  apply  the  numerical  methods 
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developed  by  Lewis*  Wang,  and  Novick  (1973)  in  connection  with  estimating 

m  group  proportions  to  o_.  present  problem.    Again,  we  will  assume  that 

all  tests  concerned  are  of  equal  length  (n^  =  n    for  all  j)  to  retain 

certain  mathematical  simplicity. 

In  the  ANOVA  terminology,  the  model  adopted  in  the  ijrevious  paper 

is  the  so-called  non-additive  model  which  includes  interaction  effects. 

Three  variance  components  <}>^,         and  ^j.  of  person,  test,  and  interaction 

effects,  respectively,  are  postulated  to  account  for  the  variability  of 

.    In  some  cases,  an  additive  model,  one  which  assumes  no  interaction 

effects,  may  be  a  satisfactory  alternative  to  the  more  general  non-additive 

model.    Specifically,  in  the  context  of  IPI,  the  students  are  tested  on 

related  skills  after  studying  the  prescribed  materials  on  a  subject.  Thus, 

the  posttest  unit  consists  of  a  set  of  tests  which  are  very  similar.  If 

the  tests  are  sufficiently  similar  (approaching  x-equi valence  in  the 

transformed  units)  so  that  the  interaction  variance  component  <J>^  is 

negligibly  small  compared  with  the  other  variance  components,  it  will 

be  adequate  to  choose  an  additive  model  in  our  analysis  (see  discussions 

in  Lord  and  Novick,  1968,  Section  7.6).    In  obtaining  joint  modal 

estimates  ^^or  Yj-*         6,,  and  6,  we  have  found  that  the  contribution 
ij      1  3 

of  individual        to  y^^  is  negligible  in  nearly  all  examples  we  analyzed. 
This  indicates  that  the  estimates  of         obtained  from  an  additive  model 
vill  be  very  close  to  those  provided  by  the  non-additive  model.    We  may 
thus  hope  the  additive  model  to  be  adequate  for  these  data.    The  advantage 
of  assuming  an  additive  model  is  that  uhe  computational  problem  in 
obtaining  marginal  estimates  for        will  be  much  easier  to  handle  while 
the  amount  of  computational  effort  ^especially  computer  time)  required 
for  this  same  purpose  in  the  non-additive  case  is  beyond  practicality. 
We  might  note  that  while  the  arc-sine  transformation  is  primarily  designed 
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for  variance  stabilization,  It  also  has  a  strong  normalizing  effect,  which 
we  have  relied  upon,  and  also  some  tendency  to  yield  addltlvlty*  Our 
findings  In  these  present  applications  seem  to  confirm  remarks  along  these 
lines  made  to  us  by  J*       Tukey*    For  these  reasons,  the  additive  case 
of  estimating  proportions  In  two-way  tables  will  be  briefly  discussed  next, 
followed  by  the  topic  of  marginal  distributions* 
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2.    An  Additive  Model  for  the  Estimation  of  Gammas 

In  this  section,  we  discuss  a  model  which  assumes  no  interaction  effect 
for  Vj^j  •    Thus,  we  may  formulate 

Vj^j  '  9  +      +  (2.1) 

where  9  =  t-i^j^lj  *  ^i  "  ^"j^lj  "  ®*  "  ^'i^lj  "  ^         defined  in  the 

appropriate  person  and  test  populations.    It  is  assumed  that  the  person 
effects  are  exchangeable  apriori  among  all  persons  in  the  population. 
Similarly,  the  test  effects  are  assumed  to  be  exchangeable  in    he  test 
population.    We  further  assume  that  the  distributions  of       and       are  normal 
and  independent  of  each  other.    Finally^  the  prior  distribution  of  d  is 
assumed  to  be  locally  uniform  within  the  range  of  interest,  and  Independent 
of  the  distributions  of       and       (see  also  discussions  in  Lindley  and 
Smith,  1972»  and  Smith,  1973).    Thus*  we  obtain  the  likelihood  and 
conditional  prior  density  of  6,  a,  and  &  as; 

liQ,  a,  6|G)  «  exp{-        ZZ  (g^^  -  6  -       -  B^)^)  (2.2) 

and 

_  m.^  _  t 

b(e,  a»  g[<^^,  <frp)  «  <fr^  ^  exp{-Za^/2(^^)  ^   exp{-i:p^/2<^g),  (2.3) 

where  G  «  contains  the  transformed  (observed)  proportions  f;^^  for 

m  persons  on  t  tests  and  the  vectors  a  and  3  contain  the  elements  and 
$y  respectively*    The  corresponding  matrix  for  the  elements  y^^  will  be 
denoted  V  .    The  zero  means  for  the  distributions  of       and       are  justified 
from  the  definitions  of  these  effects.    Note  also  that  we  discuss  only 
the  case  of  equal  length  tests  so  that  v  =  {4n  +  2)"^  is  used  in 
Equation  (2*2).    To  complete  the  model  for  our  analysis>  we  assume*  as 
usualj  independent  inverse  chi-square  prior  distributions  for  the  variances 

Er|c  34() 


*a  and  (fg,  with  parameters  (v^,  X^)  and  (Vg,  X^),  respectively  (see  Novick 
and  Jackson,  1974,  Section  7.6).  Thus, 

*g)  «  *^  exp{-X^/2.^^}  *g  exp{-Xg/2*g}  ,  (2.4) 


Combining  Equations  (2,2)  through  (2,4),  and  integrating  with  respect 
to  i)^  and        we  derive  the  posterior  joint  density  of  9,  a,  and  g  to  be: 

2  -^s(th  +  V  )  -  -Hit  +  v^) 

b(e,  a,  elO  -  [X    +  ZaU  [X    +  26,]  ^ 

i  j 

exp{-       l£:S(g     -  e  -       -  6  )^}  .  (2,5) 

Therefore,  the  joint  modal  estimates  of  9,  o,  and  g  are  found  to  be: 

e  =  g, .  (2.6a) 
"i  "  %^^i*  '  a.  =  0  (2.6b) 


where 


and 


6j  =  '^B^^'j  "  =  0  C2.6c) 


w„  =  +  t'-'-v),  tOg  =  ♦g/(*g  +  m"^)  (2,7) 


$^  =  (X^  +  Sa^)/(m  +  v^),  *g  «  (Xg  +  i:6^)/(t  +  v^)  .  (2.8) 


In  terms  of  Equation  (2.1),  we  obtain  estimates  for  the  y^^: 

*  5  +  ct^  +      =  g..  +  o^Cg^.  -  g")  +  t^gtg-j  -  g">  •  (2.9) 
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Estimates  of  proportions  based  on  Y^^  are  obtained  from  the  sine-squared 
transformations  as  described  in  our  earlier  paper < 

It  should  be  pointed  out  that  the  Y^^  are  the  estimates  of  y^j  based 
on  the  joint  modal  estimates  6,  a,  and  5  which  maximize  the  joint  posterior 
density  of  e,  a,  and  6  .    These  estimates,  Y^^,  are  not  joint  modal 
estimates  of  the  Y^j  from  the  full  joint  posterior  distribution  of  the  Y^j  • 
The  joint  distribution  of  the  mt  variables  y^^*  in  this  case,  is  degenerate 
with  actual  dimensionality  m  +  t  -  1  (<  mt).    To  see  this,  we  recall  that 
the  definition  in  Equation  (2,1),  in  effect,  shows  that  there  can  be  only 
m  -  t  -  1  linearly  independent  rows  (or  columns)  in  the  coefficient  matrix  which 
generates  y^^^  from  3,  ct,  and  Q  .    That  means,  there  e:dsts  at  least  a 
suitable  subset  SI    of  ra  +  t  -  1  elements  of  the  set  Q  *  ^Y^j)  such  that  all 
the  other  (mt  -  m  -  t  +  1)  elements  in  the  complementary  subset  n-fi    can  be 
expressed  as  linear  combinations  of  the  elements  in      ,    For  example,  given 
^11*  ^12*  ^13*         ^21  ^^^^  m  =  2,  t  =  3,  we  can  write: 


and 


=  9  +  02  +  ^2  =  Y21  +  Y12  -  > 


=  9  +  "2  +  63  =  Y21  +  Yi3  -  Yii  . 


Hence,  the  joint  density  can  be  defined  in  a  space  of  dimensionality 
m  +  t  -  1  at  most  (i.e.,  for  at  most  m  +  t  -  1  of  the  elements  Y^j)- 
However,  if  we  take  any  suitable  subset  SI   of  m  +  t  -  1  elements  from 
the  whole  set  ii,  the  joint  modal  estimates  of  the  Y^j  contained  in  SI 
and  the  variables  a.  and  6,  are  identical  to  those  y^^  obtained  from  the 
joint  distribution  of  e,  ct,  and  6  .    This  can  be  shown  by  applying 
Lemma  3.2.3  in  Anderson  (1958,  p.  47)  and  noting  that  the  set  of  variables 
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tfl  >  a. ,  3,}  is  a  one-to-one  transformation  of  the  set  {q,  a,  g}.  Thus, 


the  joint  mode  of  (il  >  a,  >  6.}  can  be  written  as; 


for  all  Y^j  contained  in      >  and 


a.  =  ES^  »  0,  B.  =  E6_j  =  0  . 

In  the  above  example,  we  can  state  that  the  joint  modes  of  y^^t  y^2* 
Yjj^t  <^*t  and  6.  are  y^^t  y^2*  ^21*  ^*  "  ^*         6.  »  0  as  having 

obtained  from  the  joint  distribution  of  9,  ot,  and  6  . 

To  illustrate  how  this  additive  modal  approximates  the  more  general 
non-additive  model  used  in  our  previous  paper,  we  have  re-analyzed  the  s^  ie 
data  with  the  present  procedure.    The  data  were  explained  in  Table  1  of 
Wang  and  Lewis  (1973).    The  same  prior  parameters  (v^  =  8,       =  .028) 
and  (^Q  *  8,       *■  .028)  used  in  the  non-additive  case  were  adopted  for  the 
present  analysis.    Estimates  of  proportions  obtaf.ned  in  the  additive  case 
are  presented  in  Table  1*    On  comparing  the  results  given  in  this  table 
with  those  given  in  Table  2  of  Wang  and  Lewis  (1973),  it  is  clearly  seen 
that  there  are  practically  no  differences  between  the  two  sets  of  estimates 
of  proportions  obtained  from  additive  and  non^-additive  models. 
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3*    Conditional  Posterior  Distribution  for  Gaimnas 
3*1    Additive  Case 

The  additive  model  described  In  Section  2  can  be  summarized  by 
the  following  statements: 

(i)    gje,  a,  §  has  a  multivariate  normal  distribution  with  mean 

vector  Y  =  A|  and  dispersion  matrix  vl,  where  V  =  (e>  a\  §*) 
is  a  vector  of  order  m  +  t  +  1>  and 


A  = 


-mt 


It 


I 


If 

it 


'7 


is  an  mt  by  m  +  t  +  1  matrix,  where  0  Indicate  zeros  In  the 

rest  of  the  matrix* 
Here  g*  and  y'  are  row  vectors  whose  elements  are  the  rows  of  G  and 

respectively*    Note  that  the  notation  1^  is  used  to  denote  a  k  x  1 
column  vector,  all  of  whose  elements  are  1  and  Ij^  an  identity  niatrlx  of 
order  k  *    The  likelihood  of  |  can  then  be  written  as 

Jt(|!g)  «  exp{-  ^  (g  -  A§)Vg  -  Ap}  *  (3*1) 

(il)  The  conditional  prior  distribution  of  |  given       and       may  be 
written  In  the  following  form: 


(3-2) 


where  D    is  the  diagona]  matrix  of  order  (m  +  t  +  1)  defined  as 
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Combining  Equations  (3.1)  and  (3,2),  the  posterior  conditional 
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distribution  of  5  given  ^    and       is  found  to  be: 

ot  p 


«  Mglp  •  b(?l*^,  *g) 


«  exp{-  ^  (g  -  A§)'(g  -  A?)  -  hVfO 

«  exp{-^  (?  -  BA'g)'B"^(|  -  BA'g)}  ,  (3.3) 

where  B~^  =  A'A  +  vD"  is  a  nonsingular  square  syrometric  matrix  of  order 
(m  +  t  +  1)  and  can  be  explicitly  expressed  as: 


,-1 


mt 


t  1 


t  1       (t  +  v/<t„)I 


•"it  hK 


1  K 

-m-t 


(m  +  v/*g)ij. 


(3. A) 


From  Equation  (3.3),  we  recognize  that  the  conditional  distribution 
of  |,  given  <|p^,  <|pg,  and  |  is  a  multivariate  normal  with  mean  vector 
(2^(?Uq»         g)  =  BA'i  and  dispersion  matrix  Var(5|<|p^,  ^i^,  g)  =  vB  . 
Thus,  the  mean  vector  and  dispersion  matrix  can  be  obtained  if  the  matrix 


B  ^  is  inverted.    Since  the  inverse  of  B  ^  is  not  easy  to  obtain  directly 
by  examining  Equation  (3,4),  we  now  try  to  find  the  posterior  conditional 
mean  vector  and  dispersion  matrix  of  ?  by  first  considering  the  conditional 
distributions  of  a,  3,  and  6  separately. 
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In  terms  of  Equations  (2.2)  and  (2,3)»  we  have 


«  /  i(e»  a,  6|g)  b(e,  a,  $\^^,  de 


«  exp{-  ^  2E  (g..-g..-a  +a.-6  +6.)^  "  21"  ^?  "  tT" 

2v  Ij  1        j  2*^  ^1     24.g  J  j 

«  exp{-  *s[v"^tE(g  .  -  g..  -  a   +  a.)^  +  'l'"^  l\V 
1  1 

•  exp{-  h[v~\  rCg.,  -  g..  -  6,  +  6.)^  +  "l-ft^  .  (3.5) 

j     j  j  6    ^  j 


Thus,  b(a,  0)  can  be  factored  into  two  parts  and  each  Involves 

only  a  or  6  .  This  implies  that  the  conditional  posterior  distribution  of 
Of  and  §,  given  and  G,  are  independent  of  each  other.    We  then 

can  write 

b(a,  bI*^,  <|.g,  G)  -  b(al<|.^,         G)  .  b(e|<|.^,         G)  .  (3.6) 

We  may  now  proceed  to  find  the  posterior  conditional  mean  vector  and 
dispersion  matrix       of  O'l'l'jjj*         ^  by  observing: 

b(a|<|.^,  G) 

a  expC-Jstv'-'-t  i;(g^.  -  g..  -      +  a.)^  +  ^a^l) 


«  exp{-  [la^  -  2R^i;a^(g^.  -  g..)  -  mR^a.  +  R^i;(g^.  -  g..)  ]} 

a 

«  exp{-  ~-  [i;(a^  -  R^g^.  +  R^g..)^  -  ^\  «?] } 


«  exp{-is(a  -  u^)*C^^(a  -  u^)}  ,  (3.7a) 
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where 


,-1 


^    [I  -—11'], 

a 


and 


R   -  A  /(<}>    +  t^^v)  . 


(3.7c) 


It  follows  that  ct|<}>  *  <}> ^  G  has  a  multivariate  normal  distribution  with 


mean  vector       and  dispersion  matrix 


vR 


R  1  1' 

T   4.     ct- m-m 

ita     m(l  -  R  ) 

a 


(3.7d) 


Similarly,  the  posterior  conditional  distribution  of         ,  ^^.f  G  is 
found  to  be  a  multivariate  normal  with  mean  vector 


and  dispersion  matrix 


;  =  — = 

^6  m 


where 


it  ^  t(l  -  Rg) 


-1 


Rg  =  +  m   v)  . 


(3.8b) 


(3.8c) 


In  order  to  obtain  the  conditional  mean  of         ,         G,  it  is  easier 

a  ts  - 

to  first  consider  the  conditional  mean  of        ,  ^.f  6,  ct  .  Since 

c(     ts    -  " 


b(e»  ^\\f  ^g,  G) 


«  /  ^(^»  a»  §|g)  b(e,  a,  6U^^»  *p)  d6 
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we  find  for  given  a, 

6 

This  Implies 

Furthermore,  referring  to  Equation  (3.7b),  we  have 

1 

R 

1 


0  . 


Hence, 


ERIC 


The  results  of  Equations  (3.7b),  (3,8a),  and  (3.9)  lead  us  to  conclude; 

=  g,.  +        -  g'-)  +  ^e^^*j  ^       *  (3.10) 

It  may  be  remarked  that  the  conditional  posterior  mean  of  Y^j  given  <{>^, 
and  G  takes  a  similar  form  to  that  of  y^^  expressed  In  Equation  (2.9)  with 
R^,  Rg  replacing  w^,  Wg,  respectively. 

To  obtain  the  conditional  variance  of  y^j  given  <{>^,  4>g,  and  G,  we 
have  to  find  the  dispersion  matrix  vB  of  V  "  (9,  ot\  6^)*    For  this  purpose 
we  make  use  of  (.he  result  A?  (^U^,  ^r^>  G)  =  BA^g  .  Since 
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(A'g)'  =  (mtg..,  tg^.,  tg^.t  ...»  tg^.»  mg.j^,  mg.j.)  , 


and 


[g..,  R^(gj^.-g..).  ...»  \ig^.-g..)f  Rg(g.j^-g..)»  ...»  Rg(g.(.-g..)] 


as  Indicated  by  Equations  (3.7b) »  (3.8a) »  and  (3.9) »  we  may  write: 


mtg., 


mg.j^ 


mg 


*t 


g. .) 
g..) 


Rg(g.l 


(3.11) 


The  elements  of  the  matrix  B  are  tuiown  except  those  of  its  first  row  and 
first  column.    Explicitly,  if  we  denote 


Cov(e»  a^U^f  *g»  G)  =  vdj 


1=1,2,  ...,m 


Cov(e,  SjU^,  <^g,  G)  =  ve^  j  =  1,  2,  t 


and 


Var(6|<^^,        G)  «  va  , 


It  Is  easy  to  verify  that 


B  -  v"-^  Var(C|4  ,  G) 


a'  ^8' 


d 


C  /v 
-a 


e' 
0 


(3.12) 


15 


where  d'      {d^y  d^),  e'  ^  (e^^,  e^),       and       are  given  in 

Equations  (3.7d)  and  (3.8b),  respectively,  and  0  is  an  m  x  t  null  matrix. 
Note  also  that  Cov(a^,  ^j^^a*  '  ^  because  the  distributions  of 

a  and  §  given        9^,  and  G  are  independent  [see  Equation  (3.6)]*  Entering 
all  the  knoim  elements  of  B  as  shown  in  Equation  (3<12)  into  Equation 
(3.11),  the  unknown  elements  a,  d^,  and  e^  can  then  be  found  by  solving; 

d^(mtg..)  tg^.  +  ^j^^^^E  tg^.  -R^(g^.  -  g..)  ,  (3.13a) 


aj(mtg..)  +^-|   mg.^+   -^^^—^  j  mg.^  =  Rg(g.j  -  g. .)  ,  (3.13b) 


and 


a(mtg..)  +  t  E  d-  g  .  +  m  E  e    g..  «  g,.  . 
i  j    ^  ^ 


(3.13c) 


It  is  easy  to  verify  that  the  solutions  to  Equation  (3«13)  are; 


(3.14a) 


and 


a  = 


nt 


R  R„ 
1  +  ^_CL^+  e 


=  v"-"-  Var(0|*^,        G)  . 


(3.14b) 


(3.14c) 


The  posterior  conditional  variances  of  y^^  given  and  G  are 

obtained  from  Equations  (3.7d),  (3.8b),  and  (3.14a)  through  (3.14c),  and 
the  fact  Cov(a^,  ^j'*ci'  *B'       "  ^' 
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=  Var(e  +      +  6^  G) 

»  Var(eU^,        G)  +  Var(a^U^,         G)  +  Var(6jU^,  G) 

+  2  Cov(e.  a^U^,         G)  +  2  Cov(e,  &  G)  +  2  Cov(a^.  6  (Jg.  G) 


nonnal  with  mean  and  variance  given  In  Equations  (3*10)  and  (3>15), 

respectively*    This  follows  from  the  definition  of  the  y.^  as  linear 

Ij 

combinations  of  and  B  [see  Equation  (2*1)])  and  the  result  that 

9,  a,  bU^jj*  <i*g>  G  has  a  (m  +  t  +  l)-varlate  normal  distribution  [see  Equation 
(3*3)].    In  passing,  we  may  note  that  similar  results  of  the  conditional 
means  of  9,  cf,  and  6  given  (J)^,  <}>g,  and  G  have  also  been  derived  by  Lindley 
and  Smith  (1972,  Section  3*1),  in  connection  with  a  general  two-factor 
design  without  Interaction* 

3-2    Non-addltlve  Case 

In  the  non-addltlve  case,  the  joint  distribution  of  y  Is  nondegenerate 

with  dimensionality  mt  *    This  can  be  seen  by  examining  the  dispersion 
it 

matrix  C    of  y  given  9,  (J)^,         and       under  the  assumptions  made  In  the 
earlier  paper  by  Wang  and  Lewis  (1973): 


(3.15) 


As  for  the  distributions  of  Yj^j  U^^^,  (J)g,  G,  they  are  each  known  to  be 
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1 

4- 
I 
I 


(3.16) 


The  matrix  C    is  a  nonslngular  symmetric  matrix  of  order  mt  .    In  contrast, 
the  dispersion  matrix  C  of  y  given  6,  ^i^,  and  ^ig  In  the  additive  case  as 
sumnarized  fr<Mn  the  assumptions  made  in  Section  2, 


C  = 


g-t  ^a-t-t 


,      B  t  , 


I 


I 


(3.17) 


is  singular,  of  rank  m  +  t  -  1  (<  rat).    Hence,  as  mentioned  earlier,  the 
joint  distribution  of  y  In  this  case  is  degenerate  with  actual  dimensionality 
m  +  t  -  1. 

As  a  result  of  the  above  distinction,  the  posterior  conditional 
distribution  of  y^^  given  <{>^,  ^^g,  ^f^y  and  G  1"        non-addltlve  case  cfia 
be  approached  in  a  somewhat  different  way.    Under  the  assumptions  made, 
the  prior  distribution  of  i  can  be  described  in  two  stages: 

(i)    Given  6,  ^i^,  ^^g,  and  ^f^y  t  Is,  aprlorl,  assumed  to  have  an 

mt-variate  normal  distribution  with  mean  vector  61  and  dispersion 
matrix  C    displayed  In  Equation  (3.16).    Note  that  for 
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convenience^  we  have  used  1  without  subscript  to  denote 

an  Hit  X  1  vector  of  ones  (viz,  1  =  1^^^)^ 
(11)  Aprlorl,  d  has  a  uniform  distribution. 
Combining  these  two  statements  with  the  normal  likelihood  of  Y  given 
It  can  be  shown,  after  a  little  algebra,  that: 

«  /  e3cp{-  ^  (x  -  s)*(y  -  g)  'h(y  -  ei)*c*"\Y  -  ei)}  de 

«  exp{-Js(Y  -  v"^B*g)*B*"^(y  -  v"V|)}  ,  (3,18) 

where 

B    ^-v^I^+C    "--C    ^Id'C    ^1)  "-1*0  ^ 

Is  an  mt  x  mt  nonslngular  symmetric  matrix.    Thus*  we  conclude  that  the 
posterior  conditional  joint  distribution  of  Y  given  4>  ,         <}^j.,  and  G 

"  ct       p       0  *' 

is  a  multivariate  normal  with  mean  vector  a    =  v    B  g  and  dispersion 

it 

matrix  B  . 

Knowing  the  distributional  form  of  )t\^^t  ♦g*         G,  we  now  proceed  to 

it  it 

find  a    and  B   without  actually  carrying  out  the  matrix  Inversion  of 
B*~^.    Applying  a  lemma  to  be  given  later  In  Section  4,  and  referring  to 
Equation  (2.11)  In  Wang  and  Lewis  (1973,  p.  7),  we  can  derive  by 
Integrating  the  expression  w.r.t*  6,  ct^  and  §i 

m  -  1                      t  -  1 
•  exp{  ^^-.j         I  (y  .  -  Y..)^  i        Uy-.  -  Y..)^> 
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Since  we  have  shoxm  that  fc(y|<t>^>  <}>^>  <t>^»  g)  is  a  multivariate  normal 
density,  the  conditional  mean  of  Y^^  given  <(>^,  <(>^,  <(>^,  and  g  can  be  found 
by  solving  the  set  of  equations; 

9  Jin  b(YU^»  <fr3»         l^/^^ij  "  0 

for  Y^j  .  After  some  lengthy  algebraic  manipulations,  this  procedure  leads 
to  the  resujt 

(3.20) 

where 

=    ,    R   =   ;   >  and  R^j  =   


■jc    »     '^rt   1  »  ft  -3 

<frg  +  V  <fr^  ^'  t^-^Wg  +  v)  ^     <fr^  +  m  \4>5  +  v) 


(3.21) 

To  obtain  the  conditional  variance  of  y,,  given  <^  ,         <tjc,  and  g, 
we  make  use  of  the  result  <^(y\^^%  <t>^,  <t>g,  g)  -  v"'''B*g  .    This  implies 
that  the  mt  diagonal  elements  of  the  matrix  v        are  the  coefficients  of 
g^j  In  expressing  C'^yi^\^a*  ^fj*  terms  of  the  elements  of  g  . 

Thus,  we  find  from  examining  Equation  (3.20) 

R          R           1  **  R    ~  R 
Var(Y,^IV  Z)  =  -i\^a'\)(f  ^   ^  ^  ^  )]  . 

(3.22) 

because  Equation  (3.18)  indicates  that  the  diagonal  elements  of  the  matrix 

it  I 

B    are  the  variances  of  Y^j|<t>(^>  <t>^»  <t>^»  |  •    In  summary  then,  we  have 

shown  that  the  posterior  marginal  distributions  of  Y^j>  conditional  on 
<t>Q»  <t>^»  are  normal  with  means  and  variances  given  by  Equations 

(3.20)  and  (3.22),  respectively.  ' 
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4.    Posterior  Marginal  Mean  Estimates  for  Gaimnas 

Having  obtained  the  conditional  means  and  variances  of  Y^^  given  the 
variance  components  <[>g>  and  we  lacw  proceed  to  find  marginal 

mean  estimates  of  gammas  by  the  same  procedure  used  in  Lewis^  Wang^  and 
Novick  (1973).    We  shall  restrict  ourselves  primarily  to  a  consideration 
of  the  additive  case  as  we  have  not  yet  been  able  to  develop  a  practicable 
numerical  algorithm  for  the  non-additive  case. 

4.1    Additive  Case 

4.1.1    Posterior  Distributions  for  the  Variance  Components 

To  obtain  the  posterior  joint  distribution  of  ^  >         consider  the 
function 

b<e»  £f»  §k^^»  ^Q*  g)  -  ^Ce»       Big)  *  b(6»  Sf»  bU^^*  ^q>* 

then» 

b(e,  cfU^,  g) 

=  /  b(e,  a,  gU^,  <tg,  g)  dg 
_  1        ^  m 

ct  1  ij 

•  exp{-     Z  <g..  -  e  -  o.)2/<<tg  +  m~'^v)}  .  <4.X) 

Thus» 

=  /  b<e»  aU^»  <tg»  g)  de 
_  izl      _  ™ 

Ot  1  Ij 

•  exp{-  h  Z  <g.,  -  g..)2/<<t-  +  m'-'-v)}  .  <4.2) 
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Also, 


=  /  b(e,  a,  6^^,  <^g»  g)  de 

_  m  _  t_ 

-       ^  exp{-        lab  '  ^  exp{-  —  26^ 

Since 


(4.3) 


and 


^^ig.,-g..~a  -Hi.-e  +e.)^  =  2z(g  -g.  -a +a.)^  +  mz(g.,-g. .-e,+e.)^  , 


we  find  from  Equations  (4.2)  and  (4.3): 
t  -  1  _ 

1  -  n^^+m-K)    2  ^  exp{-^2e^  -^2(8.j  -  8-  "  6^  +  6.)^} 

2  _i 
j      ^  ^ 

Rearranging  the  above  equation,  and  making  the  replacements  (*>g  ^  x,  v  =  c, 
m  =       t      jt,  and  n'  =  (n^^*  n^^)  =  6\  we  arrive  at  the  following 

leimoa: 

Z 

"212k  2 

/  X      exp{-  2^      -  —  s(g.j  -  g..  -     +  n.)  }  dn 

A  -  1 


«  (x  +  rt)        ^      exp{  —   Z(g.    -  g..)^}  . 

2  (x  +  k    c)    j  ^ 


(4.4) 
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Applying  the  above  lemma  to  Equation  (4.2)  and  noting  tiiat 

j:j:<8i.-8..-a  +ci.)^  =  2i:(g  -g  .-g.  +g..)^  +  t  s<g..-g..-ci,+ci.)^  . 

it  is  easy  to  verify 

///  b(e,  ct,  6|4>^,  4>g,  |)  de  da  d6 

=  /  t(aU^,         g)  da 

t  -  1  _  m  -  1 

«  (4>^  +  in"^v)       ^       <4>^  +  t~^v)  ^ 

*  exp{  ^  i —  i:<g..  -  g..)^  i —  i:(g  .  -  g..)^} 

2(4,g  +  m  \)  j      ^  2(4.^  +  t  \)  i 


<4.5) 


It  is  further  observed: 


b<4>^.  4>g!g)  =  ///  b<e.  ct,  6|4>^,      §)  •  b(4,^,      de  da  d6 

=  t<*^.        ///  t<e.  a,  e!^.^,  4>g,  g)  de  da  d6  ,  <4.6) 

where  b(4>  ,  ifr-)  is  given  in  Equation  (2.4).    Consequently,  upon 
a  p 

substituting  the  expressions  in  Equations  (2.4)  and  (4.5)  into 
Equation  (4.6),  we  have  shown 

where 

-  *"  -%(v   +2)  \  ^ 

(4.8a) 

and  f.  1 

-1    "  '^^^B     2>  ^  1 

b(4>g|g)  «  (*g  +  m    v)  .  exp  {-  _i     -  2r  ^  ' 

^  2  (4»^  +  m    v)  p 

3V)3  <4.8b) 
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2  2 
with  S^j  =  2(g.  •  -  g-.)    and  S_  =  2(g,.  -  g.  0  ,    The  fact  that  b(<fr  ,  (fr^lg) 

can  be  factored  into  the  product  of  b{*  |g)  and  b(<fro|g)  shows  that  the 

posterior  distributions  of  <fr^  and  <frg  are  independent  and  their  density 

functions  are  given  in  Equations  (4.8a)  and  (4,8b),  respectively.    It  may 

also  be  pointed  out  that  the  mathematical  forms  of  b(<fr  |g)  and  b(<fr^|g)  are 

similar  to  that  of  b(<frp|g)  in  the  m-group  proportion  case  [Lewis,  Wang, 

and  Novick,  1973,  Equation  (2.2),  p.  6]. 

4. If 2    Posterior  Marginal  Means  and  Variances  of  Gammas 

Having  discussed  the  conditional  posterior  distributions  of  y  .  given 

1 J 

<fr  ,  <fro,  and  g  in  Section  3*1  and  the  posterior  distributions  of  4  ,  in 
cx      p  -  CX  p 

if 

Section  4.1.1,  the  posterior  marginal  mean  Y^^j  of  y^j  can  be  readily 
computed  as  t 


(4.9) 


where  f>^-  ^>  ^  \t       =   S ^         and  R^,  Rg  are  defined  in  Equations  (3.7c) 
and  (3.8c),  respectively. 

The  marginal  variance  of  y^^  can  also  be  obtained  by  using  the  relation; 

Var(Y^jll)  =  C^^  ^'4>gf^^'^^^ijl*a'  •**  ^  ^'  ^^ljl*a'  * 


(4.10) 


where  the    Var     notation  is  used  to  denote 


Var  {C  (w|x»  y)]  =  SJ[C  (w|x»  y)  -         g    4^(wlx»  y)]^f(x»  y)  dx  dy  . 


x»  y 
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From  Equation  (3.15)»  we  obtain: 


(\  K  1  -  R  -  R„ 
/     £    [vc-^  +  -i  +   «  


The  second  term  in  Equation  (4.10)  Is  also  easy  to  obtain: 
Var      ill  CY^jl*„,  g)] 


Var      [g..  +  R  Cg..  -  g..)  +  RftCg.,  -  g..)] 


Cg^.  -  g..)^Var  R   +  (g.,  -  g..)^  Var  R„ 


=  (g^.  -  g..)^[^!^       -  P^]  +  (g.j  -  g..)^t(^.4,  ^6  -  Pg]  .  (4.12) 

Hence,  we  have  reduced  our  problem  to  numerical  computations  of  the  values  of 
0      1  /I  2 

p  *  Pg,  A  \>  ^nd  A  •  The  integration  problem  here  is  closely  related 
to  that  dealt  with  by  Lewis,  Wang,  and  Novick  (1973).    Thus,  the  same 


integration  algorithm  described  there  can  be  adopted  for  the  present 
applications. 

In  general,  we  are  interested  in  providing  estimates  for  the  proportions 
Tr^_j  .    This  objective  can  be  accomplished  by  applying  the  sine^squared 
transformations  to  y^^: 

=  (1  -f )  3ln\.  -  ^  ,  (4.13) 

(see  Novick,  Lewis,  and  Jackson,  1973).    A  numerical  example  will  be  given 
in  Section  5  to  illustrate  the  estimation  procedure  outlined  in  this  section. 
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4.1.3    Approximations  to  Marginal  Probabilities 


In  theory,  the  posterior  cumulative  probability  that        is  less  than 
or  equal  to  some  value  ir^  can  be  computed  by 
prob(ii^^  <  n^lg) 

-  prob(Y^^ 


f-Yo 


r 


o  J 


prob(z  <  z^)b((fr^lg)  •  b((fr^|g)d(fr„  d(fr^  , 


(4.U) 


where      =  sin  ^v^iT"  ,  z  is  a  standard  normal  varlate  and 


z  - 
o 


L  \  t  m  mt  ^  _ 


(4.15) 


However,  In  practice,  It  is  very  time-consuming  (beyond  reasonable  time 
limit  with  the  algorithms  we  have  tried)  to  evaluate  this  probability. 
We  thus  suggest  a  less  ideal  approach  which  is  an  extension  of  the  result 
in  Lewis,  Wang,  and  Novick  (1973),    There  it  was  found  that  the 
posterior  distribution  of  y^,  given  g,  can  be  satisfactorily  approximated 
by  a  normal  distribution  with  mean  and  variance  equal  to  the  posterior 
margixial  mean  and  variance  of  y,^,  respectively.    We  venture  to  generalize 
this  normal  approximation  to  the  present  case.    That  is. 
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prob{2  <  [y^  (Yij|g)]/[Var(Y^^|g)]^} 

will  be  evaluated  to  approximate  prob(Tr^j  i^^ls)*    While  this 
approxijnation  may  not  be  as  accurate  as  in  the  original  application 
to  the  m-group  proportion  case,  it  should  be  sufficiently  precise  for 
deciding  whether  a  student  should  be  advanced  to  the  next  unit  in  IPX, 
provided  the  probability  being  estiinated  is  not  in  the  extr^ne  tails. 

4.2    Non-additive  Case 

Following  the  same  procedure  employed  in  Section  4.1.1,  the  posterior 
joint  distribution  of  <^^,         and  "fr^  for  the  non-additive  model  can  be 
derived  using  Equation  (4.4)  given  in  the  lenma: 


«        +  v)"*^^"'"^'^^"^^*    +  t"^K  +  V)]      2    '         +m"^4»   -fv)]  ^ 


m-1  -  t-1 

*  +  m  ^(4 „ 


(4.16) 


2(4»^-fv)      2[*^+t  "-(^^-W)]      2[4»g4in  i^^^)] 


where 


2 


S„  "  E(g.    -  g..)  , 


and  b(<f  ,  <fr^,  (frj.)  is  a  product  of  three  independent  inverse  chi-square 
ot      p  o 

densities  with  parameters  (v^,  X^),  (v^,  X^),  and  (v^,  X^),  respectix-ely. 

It  may  be  noted  that  a  similar  result  has  also  been  obtained  by  Box  and  Tiao 
(1973,  p.  331)  in  their  discussions  of  random  effects  ANOVA. 
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In  contrast  to  the  case  of  additive  models  it  is  found  from  the 
Expression  (4.16)  that  the  posterior  distributions  of  the  three  variance 
components  are  not  independent  of  one  another.    Consequently^  triple 
integrations  are  required  tc  obtain  posterior  marginal  means  and  variances 
for  Yj^j  '    It  appears,  from  our  empirical  experience,  that  the  computer 
time  needed  for  a  triple  integration  of  a  function  of  the  form  in  Equation 
(4.16)  is  at  least  the  cube  of  what  is  needed  for  a  simple  integration  of 
the  function  of  the  form  In  Equation  (4.8a)  or  (4<8b).    Unless  ^ome 
efficient  approximations  to  these  triple  integrals  can  be  found,  this 
technique  will  not  be  practical  for  applications.    Since  we  have  not  been 
able  to  devise  an  efficient  algorithm  which  would  complete  this  analysis 
with  reasonable  cost>  we  will  not  further  discuss  it. 
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5.    A  Numerical  Example  for  the  Additive  Model 

The  data  presented  In  Table  1  of  an  earlier  paper  (Wang  and  Lewis, 
1973)  are  used  for  Illustrative  purposes  here.    Again,  we  choose  v   =       =  8 
and       ~       ^  -028  to  characterize  our  prior  distributions.    There  are 
25(-m)  persons,  5(=t)  related  tests  and  each  test  consists  of  8(*n)  items. 
As  indicated  in  Sections  1  and  2,  we  feel  that  it  is  not  far*^fetched 
to  assume  an  additive  model  for  the  analysis  of  these  data. 

The  posterior  marginal  means  and  standard  deviations  of  Y^j  given  g 
obtained  from  the  procedures  described  in  Section  A.l  are  given  in  Table  2 
(the  figures  enclosed  in  parentheses  are  standard  deviations).    It  is 

found  that  both  p    =  .7157  and       »  .7U0  (weights  used  in  computing 

ot  p 

marginal  mean  estimates  Yjj  of  Y^J  are  larger  than  03    =  .5444  and 

(Og  5=  .6637  (the  corresponding  weights  for  obtaining  the  estimates  Y^^  of 

Yj*  based  on  posterior  joint  modes  of  9,  Of,  and  6),  respectively.  From 

Equations  (2.6b),  (2.6c),  (3.7b),  and  (3.8a),  we  find  that  marginal  mean 

estimates  of  ot,  and  6,  are  accordingly  less  regressed  to  their  averages 
i  j 

(zero)  than  the  joint  modal  estimates.    The  smaller  regressions  of 

and  6^  in  this  case  result  in  discrepancies  between  Y^j  and  Y^^  .  The 

directions  of  these  discrepancies  depend  on  the  signs  of  estimated  person 

effect  and  test  effect  (which,  in  turn,  are  decided  by  the  signs  of 

g^.  -  g. .  and  g.^  -  g. .).    Specifically,  if  both  g^.  -  g. .  and  g*^  -  g.. 

are  positive  (or  negative),  y-  will  be  larger  (or  smaller)  than  Y^j  • 

On  the  other  hand,  if  g^.  -  g..  and  g.^  -  g..  are  of  opposite  signs,  their 

relative  absolute  values  will  decide  the  direction  of  the  discrepancy  and 

no  general  conclusions  can  be  mad^. 

The  estimates  Tr^,^  of  proportions        based  on  marginal  estimates 

Si 

Y^j  of  Y^j  ^T^^  presented  in  Table  3.    It  is  seen  that  there  are  sizeable 
discrepancies  between  some  jf^^  (based  on  t^j)  and         .    For  instance,  the 
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estimates  of  proportion  i^2q    3        person  20  on  test  3  are  ff^^  «  ,60r  and 

=  .548.    To  explain  this  difference,  it  is  noted  that  g^.  =  ,784  and 
ij    '  1 

g.^  =■  I1O39  (1  =  20,  j  ='  3)  are  both  smaller  than  the  overall  average 
g. .  =  1.099  in  this  case.    Therefore,  the  estimated  person  effect  and  test 
effect  are  negative.    It  follows  that        is  considerably  smaller  than 
S     because  p    >  w   and  po  ^  to^  . 

In  closing,  an  example  of  applying  the  proposed  normal  approximation 
to  marginal  probabilities  is  given  below.    From  Table  2,  we  find 

t  <^20,  3'^^  "  ^^^^^^^20,  3'?^^*^  "  person  20  on  test  3. 

pose  we  are  interested  in  a  critc 
explanations  in  Section  4.1.3,  we  obtain 

=  1  -  prob(Y2Q^  3  1  .991) 
»  1  -  prob<z  <  2.165) 
=  .0152 

-1^  \  -  $><yiM   -^^^ " 


Suppose  we  are  interested  in  a  criterion  mastery  level  ti^      i70.  Following 


<Y    =  sin    ^7  =  .991  and  —   ^  ^   ="  2.165). 

^ij 


" tVar<Y,Jg)]*^  .0739 


Both  the  estimates  S.,  ^  ,548  and  if,,  =  ,608  are  less  than  ,70  and  the 

ij  ij 

posterior  probability  that  1^20    3  ^®  greater  than  .70  is  very  small. 
Thus,  for  most  reasonable  loss  ratios,  the  action  would  be  to  retain 
this  student  in  the  old  unit  of  instruction.    For  refertitice,  approximate 
posterior  probabilities  of  tf^^  >  ,70  given  s  are  pres^ented  in  Table  4, 
In  this  table,  we  find  for  person  2,  prob(7r2j^  >  '*70|g)  *■  ,666,    Thus,  we 
may  decide  to  advance  him  on  the  basis  of  a  loss  ratio  2/1,    Inspections  of 
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Table  1  and  3  tell  us  both  112^^  =  #761  and  tS^j^  "  *7A2  are  above  the 
criterion  .70.    As  anoth-er  example,  we  find >  for  person  23  on  test  4> 
both  4    *^  *737  and  ^23    4  ^  are  above  .70.    But  we  have 

prob(Tr22    4  ^  '70||)  "  as  given  In  Table  4.    Therefore,  we  would 

advance  him  if  the  loss  ratio  is  about  1  while  retain  him  for  any 
loss  ratio  ^jreater  than  1* 
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Table  1 


Estimates  of  Proportions  with  an  Additive  Model 


X    =  X-  = 

ft  K 

.028  V 
Test 

Subject 

1 

2 

3 

4 

5 

1 

,844 

.851 

.801 

.836 

.842 

2 

.761 

.770 

.713 

.752 

.759 

3 

.887 

.894 

.849 

.880 

.886 

4 

-.747 

.756 

.698 

.738 

.745 

5 

.820 

.828 

.776 

.812 

.819 

6 

.749 

.758 

.700 

.740 

.747 

7 

.906 

.913 

.870 

.900 

.905 

8 

.810 

.818 

.765 

.802 

.808 

9 

.844 

.851 

.801 

.836 

.842 

10 

.855 

.862 

.813 

.847 

.853 

11 

.856 

.863 

.814 

.848 

.854 

12 

.906 

.913 

.870 

.900 

.905 

13 

.940 

.946 

.908 

.935 

.939 

14 

.896 

.902 

.858 

.889 

.894 

15 

.833 

.841 

.789 

.825 

.831 

16 

.772 

.781 

.725 

.764 

.770 

17 

.749 

.758 

.700 

.740 

.747 

18 

.701 

.710 

.650 

.692 

.699 

19 

.674 

.683 

.t)22 

.664 

.671 

20 

.660 

.669 

.608 

.650 

.658 

21 

.885 

.892 

.846 

.878 

.883 

22 

.856 

.863 

.814 

.848 

.854 

23 

.746 

.755 

.697 

.737 

.744 

24 

.786 

.795 

.740 

.778 

.784 

25 

.914 

.920 

.878 

.908 

.913 

-  g")^  ■ 

.68782,  Z(g.j 

=  .00495 

«  .00703, 

«  .00232,  w 

-  .5444, 

and      =  .6637 

372 


32 


Table  2 

Posterior  Marginal  Means  and  Standard  Deviations  of  Gammas 


Test 


OUUJ  Cwu 

1 

2 

1 

c 
3 

1 

1.146( 

.0704) 

1.157(.C705) 

1.092 (.0706) 

1.136(.0704) 

1.144(.0704) 

2 

1.02K 

,0709) 

l.O32(.0709) 

.967(.0711) 

1.011 (.0709) 

1.019(.0709) 

3 

1.221( 

.0712) 

1.231(.0712) 

1.167(«0714) 

1.211(.0712) 

1.219(.0712) 

4 

1.002 ( 

.0712) 

1.012(.0712) 

,947(.0713) 

.992 (.0712) 

.999 (.0712) 

5 

l.llOl 

.0704) 

1.120(.0704) 

I. 055 (.0705) 

1.100(.0703) 

1.107(.0704) 

6 

1  004  i 

'  0711^ 

^  *  V  f  XX / 

1.015(.0712) 

.950(.0713) 

.994(v0711) 

1.002(.0711) 

7 

*  V  /   Jm\>  f 

1.266(.071S) 

1.202(.0720) 

1.246(.0718) 

1.254(.0718) 

8 

•  vt  v*r  J 

1,104(<0704) 

1,040(.0706) 

1.084(.0704) 

1.092(.0704) 

9 

'  070A^ 

1.157(.07C5) 

1.092(.O706) 

1,136(.0704) 

1.U4(.0704) 

10 

'  n706^ 

1.175(,0706) 

1.111(.0707) 

1.154(.0705) 

1.162(.0706) 

11 

1.176(.0706) 

.l.il2(.0707) 

1.156(.0706) 

1.164(.0706) 

12 

1  256 1 

'  07 18^ 

1.266{,0718) 

1.202(vC720) 

1.246(.0718) 

1.254(.07l8) 

13 

1.326< 

;.0735) 

1.337(.0735) 

1.272(.0736) 

1.316(.0735) 

1.324(.0735) 

14 

X  *  ^     u  < 

'  0714^ 

1,247(.0715) 

i.l82(.0716) 

1.226(.0714) 

1,234(.0714) 

15 

1.129{ 

:,0704) 

1.140(.0704) 

1.075(.0706) 

1,119(.0704) 

r.  12 7 (.0704) 

16 

1.038< 

..0707) 

1.04S(.0708) 

.983(,0709) 

1.028(.0707) 

1.035(.0707) 

i 

1.004< 

:.0711) 

1.015(.0712) 

.950(.0713) 

.994(.07U) 

1.002(.0711) 

I  18 

.939( 

:.0723) 

.949(.0724) 

.885(,0725) 

.929(.0723) 

.937(.0723) 

19 

.903< 

:.0732) 

.913(.0733) 

.849 (.0734) 

.893(.0732) 

.901(,0732) 

20 

.885( 

;.0738) 

.895 (.0738) 

.831(.0739) 

.875(.0/38) 

.883(.0737) 

21 

l.217( 

;.07ii) 

1,227(.0711) 

i.l62(,0713) 

1,207(.07U) 

1.214(.0711) 

22 

1.166< 

:.0V06) 

1.176(.0706) 

1.112(.0707) 

1.156(.0706) 

1.164(.0706) 

23 

l.OOH 

;.0712) 

1.011(.0/12) 

.946(^0713) 

.991(.0712) 

.998(.0712) 

24 

1.058( 

;.0705) 

JL,069(. 07061 

1.004 (.0707) 

1,048(.0705) 

1.056(.0705) 

25 

1.272( 

;.0721) 

1.282(.072j) 

1.217(.0723) 

1.262(.0721) 

1.269(.0721) 

Standard  deviations  are  given  Ik  pareunheses.    Prior  Specifications: 


»  «  .028>  «  «  8j  Sample  Stat-lstijSt  )'.(g^.  -  g..)  *  .68782, 
i;(8.j  -  S-)^  -  .00495;       -  (f^^R^  «  .7157,       »  g^^^  -  .7140» 

Var  R    "  .00492,  aud  Vav  R>,  "  .00712. 
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Table  3 

Posterior  Marginal  Estiirates  of  Proportions  Based  on  ^iy^^\s) 


Test 


Subi  ect 

1 

2 

3 

4 

5 

1 

.851 

.859 

.806 

.843 

.849 

2 

.742 

.751 

.689 

.732 

.739 

3 

.907 

.913 

.867 

.900 

.905 

4 

.723 

.733 

.669 

.713 

.721 

5 

.821 

.829 

.773 

.812 

.819 

6 

.725 

.735 

.672 

.716 

.723 

7 

.930 

.936 

.893 

.923 

.928 

8 

.808 

.816 

.759 

.799 

.806 

9 

.851 

.859 

.806 

.843 

.849 

10 

.865 

.873 

.821 

.858 

.864 

11 

.867 

.874 

.823 

.859 

.865 

12 

.930 

.936 

.893 

.923 

.928 

13 

.969 

.974 

.939 

.964 

.968 

14 

.917 

.924 

.879 

.910 

.915 

15 

.837 

.846 

.791 

.829 

.835 

16 

.757 

.766 

.705 

.747 

.755 

.726 

.735 

.672 

.716 

.723 

E       1 Q 

t  DDI 

.0/1 

t  DUD 

19 

.624 

.634 

•  .567 

.613 

.621 

20 

.605 

.616 

,548 

.595 

.603 

21 

.904 

.911 

.864 

.897 

.902 

22 

.867 

.874 

.823 

.859 

.865 

23 

.722 

.732 

.668 

.712 

.720 

24 

.776 

.785 

.724 

.767 

.774 

25 

.939 

.945 

.904 

.933 

.938 

Prior  Specifications:        »  X.    =  -028^       ^  ' 
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Table  4 

Approximate  Posterior  Probabilities  of  n..  >  .70 


Test 


Subj  ect 

1 

2 

3 

4 

5 

1 

,986 

,991 

.924 

.980 

,985 

2 

.666 

.716 

.368 

.613 

,654 

3 

.999 

1.000 

.993 

.999 

.999 

4 

.559 

.615 

.270 

.503 

.546 

5 

.334 

.966 

.818 

.938 

,951 

6 

,574 

.629 

.282 

.518 

.561 

7 

1.000 

1.000 

.999 

1.000 

1.000 

8 

.928 

.946 

.755 

.907 

.924 

9 

.986 

.991 

.924 

.980 

.985 

10 

.993 

.995 

.954 

.990 

.992 

11 

.993 

.996 

.956 

.990 

.993 

12 

1.000 

1.000 

1.000 

1.000 

1.000 

13 

1.000 

1.000 

1.000 

1.000 

1.000 

14 

1,000 

1.000 

.996 

1.000 

1.000 

15 

.975 

.982 

.883 

.966 

.973 

16 

.744 

.789 

.456 

.697 

.734 

17 

,574 

.629 

.282 

.518 

.561 

18 

.  235 

.281 

.071 

.  195 

.225 

19 

.114 

.144 

.026 

.090 

.108 

20 

.075 

.097 

.015 

.058 

.071 

21 

.999 

1.000 

.992 

.999 

.999 

22 

.993 

.996 

.956 

.990 

.993 

23 

,554 

.610 

.266 

.498 

.541 

24 

.829 

.864 

.572 

.791 

.821 

25 

1.000 

1.000 

.999 

1.000 

1.000 
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Introduction 

The  BASIC  (Beginner's  All-purpose  Symbolic  Instruction  Code) 
programming  language  is  a  mathematically-based  conversational  problem^ 
solving  language,    it  has  wide  application  in  business^  scientific^ 
and  educational  environments.    It  is  powerful^  efficient^  flexible^ 
and  has  the  precision  necessary  for  most  tasks.    Also^  its  syntax 
is  simple  and  easy  to  learn.    The  BASIC  programiuing  language  is  simple 
enough  so  that  an  inexperienced  programmer  can  use  it  and  has  enough 
^ower  and  flexibility  so  that  the  experienced  program^ioer  can  write 
his  programs  efficiently.    BASIC  was  first  developed  under  Professors 
John  6.  Kemeny  and  Thomas  £•  Kurtz  at  Dartmouth  College  in  1963^1964. 
Since  then,  BASIC  has  been  transformed  into  more  than  forty  different 
major  dialects.    Each  of  these  transformations  has  added  to  or 
modified  the  original  Isrj^uage. 

Due  to  the  many  differences  among  dialects  of  BASIC ^  unless  care  is 
taken  in  the  initial  programming  it  is  both  time  consuming  and  difficult 
to  readily  translate  a  program  from  one  dialect  to  another.  However, 
if  a  few  rules  are  followed,  it  may  be  possible  to  translate  within  a 
large  set  of  dialects  with  a  minimum  of  effort.    In  this  paper  we 
investigate  this  possiblity  in  some  detail. 

The  research  reported  herein  was  performed  pursuant  to  Grant  No. 
OEG-0-72-0711  with  the  Office  of  Education^  U.  S.  Department  of  Health, 
Education^  and  Welfare^  Melvin  R.  Novick,  Principal  Investigator. 
Contractors  undertaking  such  projects  under  Government  sponsorship  are 
tncouraged  to  express  freely  their  professional  Judgment  in  the  conduct 
O  .        of  the  project.    Points  of  view  or  opinions  stated  do  not>  therefore,  381 
JC         necessarily  represent*  of ficlal  Office  of  Education  position  or  policy. 


The  specific  purpose  of  this  ctudy  is  to  Identify  and  iiwcstigiite 
in  detail  those  BASIC  dialects  that  would  form  a  set  in  which 
translatability  would  be  high  if  reasonable  programming  restrictions 
are  Imposed.    Only  BASIC  dialects  that  are  interactive  have  been 
studied.    The  following  BASIC  dialects  are  examined  in  this  study. 

BASIC  FOUR  BUSINESS  BASIC 

BURROUGHS  2500  BASIC 

BURROUGHS  5500  BASIC 

BURROUGHS  3500  BASIC 

BURROUGHS  6700  BASIC  (University  of  California  San  Diego) 

CDC  6600  KRONOS/BASIC 

CDC  6600  SCOPE/BASIC 

COM-SHARE  BASIC 

COM-SHARE  NEWBASIC 

DARTMOUTH  BASIC  (sixth  version) 

DATA  GENERAL  EXTENDED  BASIC 

DEC  PDP/8  BASIC  (EDUSYSTEM  25  and  50) 

DEC  PDP/10  BASIC 

DEC  PDP/11  BASIC 

GE  HARK  I  BASIC 

GE  MARK  II  and  GE  MARK  III  BASIC 

GE  255  riME-SHAP.IKG  BASIC 

GENERAL  AUTOMATION  ADVANCED  BASIC-16 

HONEYWELL  200  BASIC 

HONEYWELL  AOO  VEASIC 

HONEYlffiLL  316,  516,  and  716  BASIC 

HONEYWELL  600  liASIC  ^ 
HONKYWELL  16A0  XBAStC 
'    HP'aoOOB  BASIC  no-. 
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HP  2000C  BASIC 
HP  2000E  BASIC 
HP  2000F  BASIC 
HP  3000  BASIC 

IBM  CPS/BASIC  (University  of  Iowa) 

IBM  ITF/ BASIC 

IBM  CALL/360-OS  BASIC 

IBM  S3  MOD  6  BASIC 

LEASCO  BASIC 

MICRODATA  BASIC 

MULTICOMP  BASICX 

NCR  CENTURY  100  BASIC  1 

NCR  CENTURY  200  BASIC 

Q-DATA  BASIC  1 

UNICOMP/COMP  16  or  COMP  18  BASIC 
UNIVAC  1100  UBASIC 

UNIVAC  1100  UBASIC  VERSION  2.0  (Mankato  State) 
UNIVAC  1100  (University  of  Maryland  Release  1.3) 
VARIAN  620  or  V73  BASIC 
WANG  2200  BASIC 
WANG  3300  BASIC 
WESTINGHOUSE  BASIC  II 
WESTINGHOUSE  BASIC  III 
XEROX  BASIC 

Due  to  the  complexity  and  needs  of  our  applications,  we  are  maialy 
interested  at  this  time  in  a  multiuser  system  supporting  a  form  of  niass 
storage.    Therefore,  many  single  user  systems  or  small  systems  such  as  the 
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PDP-8  EDUSYSTEM  5>  10,  20  have  not  been  presented-    However^  most  of 
the  techniques  discussed  here  also  apply  to  many  of  the  smaller  systems- 

Some  of  the  dialects  are  Immediately  transportable  from  one  computer 
in  a  manufacturer's  line  to  another^  e-g-,  the  XEROX  BASIC  runs  on  the 
Sigma  5>  6,  7>  8,  and  9.    Also,  several  KASIC  dialects  are  upward 
compatible  on  computers  in  the  same  line»  the  BASIC  dialect  on 

the  Hewlett  Packard  2000B  will  run  on  the  20'^OCs  2000E,  and  2CO0F.  For 
most  dialects,  some  firanslation  must  be  done  if  a  program  written  in  the 
BASIC  of  one  computer  is  to  be  run  on  a  second  computer.    This  study 
was  motivated  by  the  desire  to  produce  readily  translatable  conversational 
language  interactive  programs  for  computer-assisted  data  analysis  and 
decision  making  in  an  educational  environment.    The  conclusions  of  the 
study  will,  however,  Bpply  quite  generally  since  the  aforementioned 
applicationr  are  very  demanding  in  terms  of  text  handling  capability, 
computational  power,  and  formatting* 

Important  Programming  Capabilities 
There  are  four  programming  capabilities  that  should  be  present  if  a 
project  of  any  magnitude  or  complexity  is  to  be  undertaken*    The  first 
of  these  is  computational  ability  and  precision.    Of  the  more  than  forty- 
five  dialects  examined^  all  were  found  to  provide  at  legist  six  digits  o'l 
accuracy  and  to  support  the  basic  arithmetic  operations  plus  exponentiation, 
Some  dialects  provided  accuracy  of  up  to  15  or  16  digits.  Obviously, 
dialects  wf.th  only  six  digit  accuracy  will  not  be  useful  in  many 
scitintific  applications*    Also,  there  was  a  large  /ariance  as  to  the 

largest  and  smallest  absolute  number  allowed.    The  smallest  maximum 

37 

absolute  number  wae  approximately  10     while  the  largest  minmum  numbt-r 

-37 

was  approximately  10       except  for  the  Uestinghou.se  and  General  Automtun 
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BASICS  which  allowed  approximately  10     and  lO"    ,    For  a  translatable 
system,  the  questions  of  accuracy  and  precision  will  need  to  be  considered 
carefully.    A  system  can  only  be  translated  to  dialects  that  provide 
the  needed  accuracy. 

A  second  necessary  capability  of  any  dialect  is  that  it  has 

It 

the  ability  to  execute  a  program  of  the  desired  size.    This  may  be 
accomplished  in  several  ways.    One  method  ^.nvolves  mass  partition  size. 
That  is,  a  user  is  allowed  as  large  a  partition  as  is  necessary  for  his 
task  and  is  swapped  in  and  out  of  core  with  many  other  users.    This  method 
may  substantially  add  to  cost  and  execution  time.    Further,  when  this 
method  is  used,  a  system  that  uses  a  monitor  to  sequentially  execute 
several  program?  is  not  very  feasible,  since  all  the  programs  and  the 
monitor  must  remain  in  core.    In  these  circumstances,  the  user  would 
load  and  execute  each  program  independently.    Such  a  procedure  results 
in  a  tolerable  inconvenience* 

A  second  method  that  is  used  by  many  dialects  is  program  cliaining. 
This  method  allows  the  user  to  fit  a  very  large  program  into  a  small 
partition  by  dividing  the  program  into  small  segments  and  executing  them 
separately  in  logical  succession*    There  are  two  kinds  of  program  chaining* 
The  first  calls  for  a  complete  overlay  of  the  program  in  core,  and  the 
second,  a  chaining  in  which  the  user  may  specify  where  the  overlay  may 
begin* 

The  third  method  for  accomplishing  the  execution  of  a  large  program 
is  through  the  U8t±  of  external  subroutine  calls*    In  this  procedure,  the 
user  calls  a  subroutine  that  is  maintained  as  a  separate  file.    After  It 
is  executed,  its  core  is  released  thus  allowing  additional  portions  of 
the  program  to  be  called  into  core  without  destroying  existing  code. 
There  are_some^SIC  dialects  such  as  IBM-CPS-BASIC,  NCR-CENTURY- 10 0-BAS I C  1, 
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MICRODATA-BASIC,  Q-DATA-BASIC~1,  UNICOMP-BASIC,  VARIAM-BASIC , 

and  WESTIKGHOUSE-BASIC-II  which  only  sllow  a  fixed  area  of  core 

and  do  not  permit  the  user  any  ot  the  above  options  for  increasing  the 

size  of  the  program  to  be  executed.    These  dialects  are  inadequate  for 

most  complicated  systems. 

The  third  capability  that  a  BASIC  dialect  should  have  is  the  means 
for  accessing  and  creating  external  data  files.    Three  levels  of  file- 
capability  are  supported  by  the  various  BASIC  dialects.    One  group  of 
dialects  offer  no  data  file  support,  e.g.,  IBM-CPS-BASIC  (i;HIV  of  IOWA), 
WESTINGHOUSE-BASTC-II  and  III,  GENERAL  AUTOMATION-BASIC,  aOHEYWELL-316, 
516,  and  716  BASICS,  MICRODATA-BASIC,  Q-OATA-BASIC,  BURROUGHS-2500  and 
3500-BASICS,  UNICOMP-BASIC,  VARIAN-BASIC,  COM-SHARE-BASIC,  UHIVAC-1100- 
BASIC  (UNIVERSITY  OF  MARYLAND  V,  1,3),  and  NCR-CENTURY -100-BASIC-l . 
Presently,  the  NCR-CENTURY-2 00 -BASIC  has  no  file  capability  although  it 
is  promised  in  the  near  future.    A  second  group  of  dialects  supports  only 
sequentially  accessed  data  files.    The  latter  group  includes  IMB-ITK- 
BASTC,  IB^f-S3-MOD-6-BASIC,  GE-255-TIME-SHARING-EXTENDEl)-BASIC,  HONEYUKI.L- 
1640-BASIC,  WANG-3300  and  2200-BASICS,  IBM-CALL/OS-BASIC,  CI)C~6000 
KRONOS-BASIC,  UCSD-B(i700-BASIC ,  CDC-6000-SCOPE-BASIC,  BURROUGHS-B550O- 
BASIC,  DEC-PDP-8-BASIC  (EDUSYSTE^f  25  and  50),  and  UNIVAC-llOO-UBASIC 
(MANKATO  STATE  VERSION  2.0).    A  third  group  of  dialects  supports  both 
sequential  access  and  random  access  files-    Members  of  this  group  are 
liP2OOOF-2O00E-20OOC-200Ol?-3AS IC ,  UNIVAC-llOO-UBASIC ,  HP3000-BASIC , 
MULTICOMP-BASTCC,  BASIC-FOUR-BUSINESS-BASIC,  XDS-BASIC,  CE-HARK-I, 
MARK-Il,  AND  MARK-IIX-BASICS,  LEASCO-RESPONSE-I-BASIC,  DARTMOUTIt-BASR , 
COM-SHARE-BASIC  and  NEWBASIC,  DEC-PDP~10  and  PDP-ll-BASIC ,  HoNEYWELl.- 
200,  400,  and  600-BASlCS,  and  DAVA-GENERAL-1J/.SIC.    The  urgency  of  the 
need  for  random  access*  fiJes  varies  with  the  app1ii:atjon.    llov't'Vt,')  .  •■tiivt,' 


some  type  of  file  support  is  needed  for  nearly  all  applications^  a 
minimum  of  sequential  access  to  files  is  almost  a  must. 

External  files  ^i^e  used  to  store  data  that  are  too  complicated  and 
time  consuming  to  recompute  every  time  they  are  needed.    Files  also  are 
needed  to  pass  data  between  chained  segments  of  a  system^  if  the  whole 
partition  is  overlayed.    Also,  files  can  be  used  to  store  results  of 
computations  so  that  the  user  may  decrease  the  size  of  his  program.  Tn 
view  of  this,  the  BASIC  dialects  mentioned  in  the  second  and  third 
groups  of  the  previous  paragraph  are  more  adequate  than  the  dialects  in 
the  first  group. 

A  fourth  important  capability  for  a  BASIC  dialect  is  its 
conduciveness  to  generating  formatted  output.    This  is  accomplished  by 
means  of  the  PRINT  USING  statement.    This  statement  allows  the  user  to 
determine  what  his  output  is  going  to  look  like.    He  may  specify  the 
number  of  digits  to  be  outputted,  the  mode  of  output,  and  the  column (s) 
in  which  the  output  is  to  appear.    Also,  the  user  may  specify  carriage 
control,  e*g*»  number  of  spaces  between  lines.    Most  of  these  may  also 
be  accomplished  using  a  PRINT  statement.    This  is  much  less  efficient, 
requires  more  programming,  and  cannot  be  accomplish^.d  in  the  case  of 
specifying  the  number  of  digits.    The  PRINT  USING  statement  has  different 
syntax  In  almost  every  dialect.    Therefore,  it  should  be  noted  that  if 
the  PRINT  USING  is  used,  it  must  be  modified  when  translating  from  one 
dialect  to  another.    In  some  dialects  the  format  to  be  followed  is 
specified  in  the  PRINT  USING  statement  itself,  while  in  others  the 
format  is  in  an  IMAGE,  FIELD,  or  format  statement.    Some  dialects  uco 
Fortran  format  for  output,  e.g*,  MULTICOMP-BASICX.    Others  use  an 
example  output  line  with  special  characters  denoting  numeric  output. 
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^^ot  all  systems  have  formatted  output,  e.g.,  HP2000b**BASIC , 
HP2000E  BASIC,  KCR-CENTURY'200-BASIC,  CDC-6000-SC0PE  and  KRONOS-BASICS , 
and  UKIVAC^llOO -BASIC  (.UNIVERSITY  OF  MARYLAND).     It  is  felt  that  a 
system  should  have  a  capability  for  formatted  output.    However,  i£  it 
does  not,  the  PRINT  statement  can  provide  many  of  the  features  of  the 
PRINT  USING  coHHTiand.    Although  the  results  may  not  be  usually  as 
appealing  as  with  the  PRINT  USING  statement,  they  provide?  a  satisfactory 
alternative. 

The  translation  of  most  statements  in  a  BASIC  dialect  will  be 
trivial  or  no  translation  will  be  necessary.    Operands,  relations, 
names,  strings,  arrays,  functions,  input,  and  branching  can  be  translated 
with  little  effort  or  time.    The  thrtie  difficulties  that  will  be 
encountered  are  file  handling,  chaining  or  subroutine  calling,  and 
output  formatting.    Since  there  is  no  exact  standard  for  these  areas, 
a  knowledge  of  the  statement  formats  in  these  areas  can  help  to 
minimize  the  expenditure  of  time  and  energy. 

Comparison  of  Clements 
Operations  and  Relations; 

All  BASIC  dialects  use  the  same  symbols  for  addition  subtraction 
multiplication  *,  and  division  /.    However,  there  is  no  standard  operator 
for  exponentiation.    Different  dialects  use  the  following  symbols: 
**,  +,  ^  .    The  most  frequent  symbol  used  for  exponentiation  is  ^ 
If  exponentiation  can  be  avoided,  translatability  in  operands  is 
achieved.    The  string  operation  of  concatenation  is  not  implemented  on 
all  dialects*    For  those  in  which  it  is  Implemented,  ampersand  (&), 
plus  (+),  comma  (,),  STR,  or  CATS  are  used.    A  few  of  the  dlnlects  such 
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as  BASIC-FOUR,  VARIAN,  COM-SHARE-NEWBASIC,  HP2000B,  HP2000C,  HP2000E, 
HP2000F,  HP3000,  PDP-11,  LEASCO,  WANG-3300,  WESTINGHOUSE,  GENERAL- 
AITjOMATION,  and  the  UNIVAC-llOO  Implement  the  logical  operands  of 
AND,  OR,  and  NOT-    The  PDP-11,  COM-SHARE-NEWBASIC,  and  UNIVAC-llOO 
BASIC  dialects  also  support  logical  equivalence  (EQV,  EQU,  and  EQU, 
respectively),  exclusive  or  (EOR,  XOR,  and  XOR,  respectively),  and 

Implication  (IMP)-    GENERAL-AUTOMATION  also  supports  exclui;lve  or  (XOR) - 
The  logical  relations  symbols  for  less  than  (<),  greater  than  (>) , 
not  equal  (<>),  less  than  or  equal  (<*) ,  and  equal  («)  are  standard 
across  all  the  BASIC  dialects  except  for  the  UNIVAC-llOO-UBASIC 
(VERSION  2.0  MANKATO  STATE  COLLEGE)  dialect  which  uses  LSS  for  less 
than,  GRT  for  greater  than,  NEQ  for  not  equal,  LEQ  for  less  than  or 
equal,  and  EQU  for  equal,  and  MICRODATA  which  uses  //  for  not  equal. 
The  logical  relation  greater  than  or  equal  (>'^)  is  standard  across  all 
BASIC  dialects  except  for  the  U15IVAC-1100  (VERSION  2-0  MANKATO  STATK 
COLLEGE)  and  HONEYWELL- 200-BASIC  dialects  which  use  the  symbols  GKQ 
and  respectively- 

Names : 

In  the  BASIC  programming  language,  there  can  be  up  to  five  types 
of  variable  names.    These  are  array  variable  names,  numeric  variable 
names,  string  variable  names.  Integer  variable  names,  and  user  defined 
function  names.    A  numeric  variable  name  should  be  either  a  letter  or 
a  letter  followed  by  a  single  digit,    l^rhile  the  IBM^BASIC  dialects 
allow  the  special  characters  of  $,  @,  and  if  to  be  used  anywhere  a  letter 
may  be  used,  and  IBM"CPS^BASIC  allows  a  single  letter  or  a  letter 
followed  by  another  letter  or  a  number,  for  reaitons  of  translatability 
these  conventions  should  not  be  used*    String  variables  are  used  in 
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all  BASIC  dialects  except  NCR-CKNTURV^lOO-BASIC-1,  BURROUGnS-2500 
and  3500-BASICS,  UNTCOMP--BASIC,  VARIAN-BASIC ,  WESTINGHOUSK-BASIC-Il , 
nONEyWELL-316,  516,  and  716-BASICS,  MICRODATA-BASIC,  and  Q-DATA 
BASIC-1- 

There  are  two  conventions  used  for  string  variable  names ^  The 
first  is  a  letter  followed  by  a  $.    The  second  is  a  numeric  name 
followed  by  a  $.    For  translatability  the  first  convention,  a  letter 
followed  by  a  $,  should  be  used-    Integer  variable  names  are  only 
allowed  in  the  PDF-ll-BASIC  and  HP3000-BASIC  and  should  be  avoided- 
Array  variable  names  should  be  confined  to  a  single  letter  thai  ha« 
not  been  used  elsewhere.    Some  dialects  allow  any  numeric  name  to  be 
an  array  name  and  allow  the  same  name  to  be  both  an  array  variable  nnme 
and  a  numeric  variable  name.    In  the  interest  of  translatability, 
array  variable  names  should  be  confined  to  a  single  unique  letter- 
User  defined  function  names  are  standard  in  all  BASIC  dialects  except 
the  HCR-200-BASIC,  UKICOMP-MSIC    and  PDP-ll-BASIC .    There  are  no  u«ei 
defined  functions  in  the  NCR-200  and  UNICOMP-BASIC  dialects.  The 
PDP-ll-BASIC  allows  the  user  defined  function  to  be  FN  followed  by  dny 
numeric  variable  name.    All  other  BASIC  dialects  limit  a  user  defined 
function  to  Fl^  follov7ed  by  a  single  letter-    The  f^eaeral  convention  of 
FK  letter  should  be  used* 

Strings: 

All  BASIC  dialects  for  the  U^^ICO^^?-BASIC,  BURROUGHS-2500  and  3500- 
BASICS,  VARIAN-BASIC,  WESTINGHOUSE-BASIC-II,  IIONF:YWIvLL-316 ,  516,  and 
716-BASICS,  MICRODATA-BASIC,  Q-DATA  BASIC -1 ,  and  NCR  CENTURY-lOO- PASU  1 
have  ^string  handling  capabilities.    However,  these  dialect*^  f;tiil  .illow 
strings  in  I'RINT  Rtatement:«t    String  coii«taut«  are  encJo.sed  iu  :iuott^f** 
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In  all  BASIC  dialects  except  IBM-CPS-BASIC,  IBM-S3-MOD-6-BASIC, 
UNIVAC-llOO-'UBASIC  (VERSION  2.0  MANKATO  STATE),  and  XDS-BASIC, 
double  quotes  (")  may  be  used.    In  the  exceptions,  single  quotes 
(')  are  used.    Therefore,  if  translation  is  to  take  place  between 
dialects  that  use  the  different  types  of  string  quotes,  a  user  must 
be  sure  to  change  all  the  quotes.    Strings  vary  in  length  in  the  BASIC 
dialects.    The  shortest  string  length  is  6  characters  and  the  longest 
string  length  is  over  3^^000  characters.    Ther<!  are  two  groups  of 
dialects,  tuose  that  allow  a  maximum  of  6  to  22  characters  and  those 
that  allow  string  length  greater  than  or  equal  to  72.    The  -iialects 
that  provide  a  string  length  less  than  or  &qual  to  22  characters  arc 
DEC-PDP-8-BASIC  (EDUSYSTEM  25  and  50),  BURROUGHS-B5500-BASIC,  IBM-CPS- 
BASIC,  IBM-S3-MOD-6-BASIC,  WANG-3300-BASIC,  IBM-ITF-BASIC,  XDS-BASIC, 
IBM-CALL/360-OS-BASIC,  HONEWELL-200-BASIC,  GE-255-TIME-SHARING-BASIC, 
GE-MARK-I-BASIC,  NCR-CENTURY-200-BASIC,  and  UCSD-B6700-BASIC,  Sever^ii 
of  the  BASIC  dialects  provide  string  processing  functions  from  which 
substrings,  positions,  lengths,  and  other  data  may  be  obtained.  It 
should  be  noted  that  these  functions  are  not  translatable  and  should 
not  be  used  if  the  system  is  to  be  translated.    If  string  handling  is  not 
needed,  then  all  BASIC  dialects  can  be  considered.    But  if  a  long  string 
(greater  than  22)  is  needed,  then  translatability  is  limited* 

Arrays : 

All  BASIC  dialects  allow  use  of  arrays  to  store  data*    An  array 
may  have,  at  most,  two  dimensions  in  all  BASIC  dialects  except  CDC" 
6600-SCOPE-BASIC,  BASIC-FOUB^BUSINESS-BASIC,  VARIAH-BASIC,  CDC-6600" 
KRONOS-BASIC  and  the  H0NEYWELL-200^BASIC,  which  allow  three  dimensions, 
UNIVAC-llOO-BASIC  which  allows  four  dimensions,  and  COM-SHARE-NEWBASIC, 
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WESTINGHOUSE-MSIC,  GENERj  T--AUT0MATI0N-BASIC»  and  HOKEYWELL-4pO,  316 > 
516,  and  716-BASICS  whicU  allow  as  many  dimensions  as  will  fit  in  one 
£?tatement.    All  BASIC  dialects  Uave  some  limit  on  the  number  of  clotpeutij. 
In  IBM-CPS-BASIC  the  limit  is  500  elements  per  array.    But  in  most  BASIC 
dialects  it  is  limited  only  by  the  amount  of  core  that  is  available. 
Arrays  that  do  not  appear  in  a  dimension  (DIM)  statement  arc  dimensioned 
ten>  or  ten  by  ten>  or  ten  by  ten  by  ten  dependir^  upon  use  and  system, 
in  all  dialects  except  PDP-8-BASIC  (EDUSYSTEM  25  and  SO),  BASIC-FOUR- 
BUS  TNESS-BASIC,  GENERAL-AUTOMATION-BASIC »  WAKG-3300  and  2200*BASIC.S,  <ind 
NCR- CENTURY "200'-BASIC.    Therefore,  all  arrays  shoJld  be  dimensioned  for 
t,ranslatability,    Depending  upon  the  dialect ,  ,an:ays  start  nt  zero  or 
one.    But  in  matrix  (MAT)  operations*  the  zero> elements  arc  ignored 
anyway.    All  BASIC  dialects  have  the  MAT  operaLions  Addition*  subtraction* 
scaler  multiplication*  multiplication,  transposition*  and  inversion 
except  the  PDP-8-BASIC  (EDUSYSTEM  25  and  50),  NCR-CENTURY--200-BASIC, 
NCR"CENTURY-100-BASIC-1»  BASXC"FOUR-BUSINESS-BASIC»  UNICOMP* BASIC  * 
WESTINGHOUSE-BASIC-II*  HONEYWELL-316 ,  5l6,  and  716-BASICS*  HICKODATA- 
BASIC*  Q-DATA-BASIC*  WANn-2200-BASTC *  and  UCSD-B6 700* BASIC  which  do  noi 
support  MAT  operations.    Also*  there  is  an  identity  matrix  (IDN)*  n 
matrix  of  all  ones  (CON)  and  a  zero  matrix  (2ER)  in  all  dialects  that 
have  the  MAT  commnndjj.    All  dialects  that  support  the  MAT  commands  olsu 
support  a  form  of  matrix  input  aad  output.    In  addition*  some  ->upr^>rt 
a  file  input  and  output  for  matrices^.    UHietlier  nn  array  is  traiislutahl<? 
or  not  depends  upon  several  factors,  including  program  size  and  i>artition 
size.    The  PDP-ll-BASIC*  HONEYmL"400-BASIC *  and  COM-SUARr-BAJilC  .lilow 
arrays  to  re^jide  on  disc  in  what  is  called  their  virtual  ntorapt^. 
these  nre  the  only  dialects  that  support  a  featuro  like  this. 
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Complex  Variables: 

Only  the  HP3000-BASIC  and  COM-SHARZ-NEWBAbIC  dialects  allow  the 
use  of  complex  variables.    Therefore,  this  capability  should  be  avoided. 

Functions : 

BASIC  functions  are  divided  into  two  types.    The  first  type  includes 
all  functions  permanently  resident  in  the  syst(jm.    All  BASIC  dialects 
support  the  following  system  functions: 

ABS  Absolute  value  (except  UMICOMP-BASIC) 

ATN  Arctangent  (except  WESTINGHOUSE-UASIC-II  and  BASIC-FOUR; 

and  BURROUGHS- 2500  and  3500  and  COM-SHARE-NEWBASIC  whicii 
use  ATAN) 

COS  Cosine  (except  \;ESTIKGH0USE-BASIC-II  and  BASIC-FOUR) 

EXP  Exponentiation  (except  BASIC-^FOUR^ BASIC) 

INT  Largest  integer  (except  UNICOMP-BASIC) 

LOG  Common  logarithm  (except  BASIC -FOUR-BASIC) 

RND  Randomization  (except  WESTINGHOUSE-BASIC-II,  UNICOMP-BASIC , 

and  BASIC-FOUR-BASIC;  and  COM-SHARE-NEWBASIC  which  uses  num) 
SGN  Sign  (except  UNICOMP-BASIC 

SIN  Sine  (except  WESTINGHOUSE^BASIC-II  and  BASIC-FOUR-BUSINESfi- 

BASIC) 

SQR  Square  root  (except  BASIC- FOUR-BUSINESfi-BASTC  and  WESTTNGHOUf;t> 

BASIC  "II) 

TAN  Tangent  [except  IBM-CPS-BASIC  (UNIVERSITY  of  IOWA),  UNICOMP- 

BASIC,  BASIC -FOUR-BUSINESS-BASIC,  and  WESTINGHOUSE-BASIC-II) 

The  preceding  system  functions  can  be  used  freely  unless  in  one  of 
the  exception  dialects*    The  various  dialects  also  support  many  other 
functions  that  should  be  avoided. 
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The  second  type  of  function  is  a  user  defined  function.  These 
functions  pass  one  or  several  arguments  depending  on  the  dialect.  A}5o» 
some  dialects  allow  multiple  line  definitions.    To  be  truly  translatable^ 
only  single  line  definitions  that  pass  at  most  one  variable  should  be 
used.    All  BASIC  dialects  allow  user  defined  functions  except  for  HCR- 
CENTURY- 200 -BASIC  and  UNICOMP-BASIC. 

Branching: 

There  are  four  t;ypes  of  statements  used  In  BASIC  for  branching 
purposes.    The  first  type  of  branching  statement  is  the  FOR  statement. 
This  loops  control  between  the  FOR  statement  and  its  corresponding 
NEXT  statement  until  a  counter  reaches  a  limit.    The  format  that  is 
used  in  all  BASIC  dialects  is; 

POR  variable  =  initial  value  TO  limit  STEP  increment, 

NEXT  variable 

Initial  value*  increment  and  limit  may  be  any  expression  in  ail  BASIC 
dialects  except  BASIC-FOUR-BUSIHESS-BASIC,  and  COM-SIIARE-BASIC.  Tn 
COM- SHARE -BASIC  limit  must  be  a  number  and  in  BASIC^FOUR-BUSXKESS-BASIC 
initial  value,  increment  and  limit  may  be  variables.    In  all  BASIC 
dialects  the  loop  works  In  the  following  manner: 

1)  The  variable  is  set  equal  to  the  initial  value. 

2)  Test  if  variable  is  searched  or  passed  the  limit. 

a)  Execute  loop  if  limit  has  not  been  reached. 

b)  Exit  loop  if  limit  has  been  reached. 

3)  Add  increment  to  variable. 

4)  Go  back  to  step  2. 
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In  all  BASIC  dialects  loops  may  be  nested^  but  Tnaximuin  nesting  permitted 
varies  between  dialecCs.    If  the  user  picks  five  as  the  deapesc  loops 
can  be  nested^  then  the  system  should  be  translatable. 

The  second  type  of  branching  statement  is  t;ie  IF  statement.  There 
are  many  forms  of  the  IF  statement  in  the  BASIC  dialects;  but  there  is 
one  that  holds  acres  s  all  dialects.    Thac  iss 

IF  expression  logical  operator  expression  THEN  line  number.. 

The  third  type  of  branching  statement  is  f:he  GOTO  statement.  There 
are  two  forms  of  this  statement,  the  simple  GOTO  and  the  computed  GOTO* 
The  computed  GOTO  is  not  implemented  in  all  dialects  and  should  be  avoided. 
The  simple  GOTO  is  standard  in  all  dialects  as: 
GOTO  line  number. 

The  word  GOTO  may  also  be  GO  TO  in  some  dialects  but  it  is  not  clear 
from  the  manuals  which  is  accepted. 

The  fourth  type  of  branching  statement  is  the  GOSUB  statement. 
Here  there  are  also  two  forms,  the  simple  GOSUB  and  the  computed  GOSUK. 
The  computed  GOSUB  is  not  universal  and  should  be  avoided.    The  simple 
GOSUB  has  the  following  syntax 

GOSUB  line  number. 
This  form  is  standard  across  all  BASIC  dialects. 

Therefore,  if  the  preceding  forms  of  the  branching  statements  are 
used,  the  users*  system  will  be  translatable  in  terms  of  branching. 

Input: 

In  the  BASIC  programming  dialect  there  are  two  methods  for  accepting 
input.    The  first  method  is  the  READ-DATA  statement  pair.    Th/^se  two 
statements  are  completely  translatable  across  all  BASIC  dialects  except 
BASIC *FOUR-BUSINESS"BASIC  which  does  not  allow  READ-DATA  pairs.    The  form 
of  these  two  statements  is: 
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READ  var  1>  var  2>  ...  var  n 

DATA  constant >  constant >  ...  constant. 
The  only  restriction  is  that  in  the  NCR-CENTURV-lOO-BASIC-1 ,  BURROUGHS' 
2500  and  3500-BASICS,  COM-SHARE-BASIC,  WESTINGHOUSE-BASIC-II ,  HONEVWELl/ 
315,  516,  and  716-BASICS,  MICRODATA-BASIC,  and  Q-BATA-BASIC  do  not  ollow 
string  variables  or  constants  in  the  READ  or  DATA  statements.    The  next 
read  position  in  the  data  list  can  be  reset  to  the  beginning  using  the 
RESTORE  cornmand  In  oil  BASIC  dialects  except  DARTMOUTH-'BASIC  which  ukgs 
the  RESET  statement  and  UIJICOMP-BASIC  which  has  liC  prevision  for  starting 
over  in  a  DATA  statement. 

The  second  method  for  accepting  input       via  the  INPUT  statement. 
In  BASIC  the  INPUT  statement  accepts  input  from  the  user's  terminal.  The 
INPUT  statement  has  the  following  syntax; 

INPUT  var       var  2>         var  n. 
This  syntax  is  constant  over  all  BASIC  dialects  for  this  Rtatement>  althoutili 
the  same  dialects  that  do  not  allow  scrlngs  in  READ-DATA  pairs  do  not  billow 
strings  here.    Thus>  these  statements  are  easily  translatable. 

Files: 

The  least  translatable  of  all  the  statements  are  the  file  handling 
statements.    Different  dialects  have  different  methods  for  hjndling  liles. 
In  some  dialects  the  user  allocates  a  file  name  with  a  FILK  s.:atemenl.  >  a 
FILES  statement  or  an  ASSIGN  statement  depending  upon  the  dialec:..  Other 
dialects  implicitly  do  this  in  the  OPEN  statement  or  first  access. 
Backspacing  and  rewinding  of  files  axe  allowed  in  a  few  dialects.  Some 
dialects  read  from  files  with  an  INPUT  statement  ^'hlle  others  ufio  a 
RKAD  statement.    Also,  PRINT  and  WRITK  statement*;  are  used  for  wrUiiifi 
into  flies  in  different  dialects.    Some  dialects  sense  for  ond  of  ftJo 
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with  an  IF  END  statement,  others  use  a  NODATA  statement,  while  others  use 
an  E^FILl  statement.  File  names  are  determined  from  dialect  to  dialect 
and  even  from  installation  to  Installation  within  a  dialect.  TherofoiR, 
file  handling  is  not  directly  translatable  and  the  program  writer  should 
attend  carefully  to  file  input  and  file  output  statements  when  designing 
translatable  programs. 

Miscellaneous: 

There  are  several  aspects  of  BASIC  that  do  not  fall  into  any  of 
the  above  categories.    The  first  of  these  is  the  range  on  line  numbers 
across  the  different  dialects.    The  maximum  range  found  was  from  0  to 
99999999.    However,  all  dialects  except  IBI^CPS"BASIC  and  the  PDP-8-BASIC 
accept  line  numbers  from  I  to  9999.    IBM-CPS-BASIC  has  a  range  from  1 
to  999  and  PDP-8-BASIC  (EDUSYSTEM  25  and  50)  has  a  range  from  1  to  2046. 
Therefore,  cne  sUould  use  line  numbers  only  from  1  to  9999  for 
translatability.    Unless  either  of  the  two  above  exceptions  are  to  be  used. 

Another  feature  is  comments  or  remarks;  these  can  be  fully  trans- 
latable if  the  syntax  it*: 
REM  message. 

Some  dialects  zero  all  variables  before  they  are  used,  but  this 
should  not  be  taken  for  granted  across  all  the  dialects. 

Also,  certain  dialects  such  as  PDP-ll-BASIC,  COM-SHARE-NEWBASIC, 
HONEYWELL-316,  516,  716,  and  600-BASICS,  HP3000-BASIC,  WANG-3300,  and 
2200-BASlCS,  and  the  HONEYWELL- 200- BASIC  allow  multiple  statements  on 
a  single  line.    This  feature  should  not  be  used. 

The  keyword  tET  should  not  be  dropped  from  assignment  statements 
since  many  of  the  dialects  require  it*    Also,  only  one  variable  should 
be  assigned  at  a  time  even  though  many  dialects  allow  multiple  assignments. 
The  format  appears  as:  .  ^ 


LET  var  =  expression. 
The  following  three  statements: 
STOP 
END 

are  conipletely  translatable  when  used  in  the  above  iiynLax-    Some  dialects 
allow  a  comment  to  fcxlov.    This  should  be  avoided  for  reasons  of  trans- 
latability. 

Summary  of  Rules  for  Translatabillty 

1)  Avoid  the  use  of  exponentiation  if  possible  or  use  ^  in  all 
dialects  where  it  is  permitted. 

2)  Do  not  use  logical  arithmetic  (OR,  AND,  NOT,  etc), 

3)  Use  the  following  logical  relations^    <,  >,  <>,  =,  and 
>=  whenever  permitted. 

4)  Use  a  single  letter  or  a  letter  followed  by  n  number  for  a 
numeric  variable  name. 

5)  Use  a  single  letter  followed  by  n  $  for  string  variable  iuithl'^*;. 

6)  Use  a  unique  letter  for  an  array  variable  name. 

7)  Use  FN  followed  by  a  singlt^  letter  for  a  user  defined  funcciOn 
name. 

8)  Use  double  quotes  (")  whenever  possible. 

9)  Decide  on  what  length  strings  are  going  to  be  allowed  and 
translate  your  system  within  the  group  your  strint;  K^ngth 
specifies . 

10)  Avoid  string  handling  system  functions. 

11)  Use  at  most  two  dimension  array,';. 

12)  Start  arrays  at  1. 

13)  Take  advantage  of  the  MAT  command  where  applicable , 
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14)  Only  the  system  functions  listed  should  be  used. 

15)  Use  only  single  line  user  defined  functions, 

16)  Nest  loops  at  most  five  deep. 

17)  Limit  the  following  statements  to  the  listed  format. 
FOR  variable  =  variable  TO  expression  STEP  expression 
NEXT  variable 

IF  expression-operator-exprcssion  THEN  line  number 

GOTO  line  if 

GOSUB  line  // 

READ  var  1>  ... 

DATA  constant  1>  ... 

INPUT  var  1,  -  - - 

STOP 

END 

RETURN 

RESTORE 

REM  message 

LET  var  =  expression 

18)  Do  not  use  multiple  statements  on  a  single  line* 

19)  Line  numbers  should  run  from  1  to  9999. 

20)  Do  not  expect  the  system  to  zero  all  variables - 

21)  Avoid  integer  and  complex  variables. 

Translatable  liASIC  Dialects 
Most  of  the  problems  in  translating  one  dialect  to  another  arc  a 
matter  of  changing  a  keyword  or  format-    These  changes  can  be  made  to 
the  whole  program  at  one  time  using  the  edit  features  of  the  system. 
There  are  two  features  that  must  be  changed  or  at  least  checked  very 
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closely.    These  are  the  file  handling  nnd  foifnatted  output  capabilities. 
These  are  not  difficult  chang<;s  to  makc^  but  must  be  coiisid<;red  care- 
fully. 

It  was  found  that  the  dialects  studied  fell  into  three  catef;orie». 
The  first  of  these  categories  contains  those  dialects  that  are  missing 
a  critical  element.    These  are: 

iimiROUGHS'-2500-BASIC  (no  data  file  capability,  no  capacity  for 
chaining  etc. ) 

BURROUGHS- 3500 -BASIC  (no  ^ata  file  capability,  no  capacity  for 

chaining  etc.) 
COM- SHARE- BASIC  (manual  chaining  only) 
GEHERAL-AUTOMATION-BASIC  (no  data  file  capability) 
HONEYWELL- 316,  516,  and  716-BASTCS  (chain  only  FORTRAN  or 

ASSEMBLER  routines) 
IBH-CPS-BASIC  (03IV  OF  IOWA)  (no  data  file  capability  >  no  capaci.lv 

for  chaining  etc. ) 
MICRODATA'-BASIC  (no  dat^  file  capability,  no  capacity  for 

chaining  etc*) 

MCK-CENTURY-IOO-BASIC  (no  data  file  capability*  no  capacity  fnr 
chaining  etc.) 

MCR-CENTURY-200-UASIC  (no  dnta  file  capability  at  this  time) 
Q-DATA-BASIC  (no  data  file  capability,  no  cnpacicy  for  chainiiijx  etc. 
UNICOMP-BASIC  (no  data  file  capability,  no  capacity  for  chaining,  etc 
UfJIVAC-llOO  (UNIV  OF  MARVLAKD  VERSION  1*3)  (no  data  file  capnbilUy) 
VARIMl-BASIC  (no  data  file  capability) 

KESTINGIIOUSE-BASTC-Il  (no  data  filo  capabijity*  im  ccipacicy  for 

chaininj:  etc  - ) 
WESTINGHOCSi>BAPTC-m  (no  data  file  cnpability) 
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The  second  category  contains  those  dialects  that  only  do  not  hnve 
formatted  output  capability. 

BASIC-FOUR-BUSINESS-BASIC 

BURROUGHS-5500-BAS IC 

DEC-PDP-8-BASIC  (EDUSYSTEM  25  and  50) 

GE-255-TIME-SHARING-BASIC 

HP2000B-BASIC 

HP2000E-BASIC 

UCSD-B6700-BASIC  (UNIVERSITY  OF  CALIFORNIA,  SAN  DIEGO) 
WIIVAC -1100 -U BASIC  (VERSION  2.0  MANKATO  STATE  COLLEGE) 
Also  included  in  this  category  are  those  dialects  that  issue  mass 
storage  in  place  of  chaining  or  external  subroutine  capability. 
CDC- 6600-KRONOS- BASIC  (also  no  formatted  output) 
CDC-6600-SCOPE-lJASIC  (also  no  formatted  output) 
im-CALL/OS-360-BASIC 
IBM-ITF-BASIC 

The  third  category  contains  tho^e  dialects  which  are  preferred. 
COM-SHARE-NEWBASIC 
DATA- GENERAL-BASIC 
DARTMOUTH-BASIC 
DEC-PDP-10  BASIC 
DEC-PDP-11  BASIC 
GE-MARK-I -BASIC 
GE-MARK-II-BASIC 
GE-MARK-I II -BASIC 
HONEYWELL- 200- BAS IC 
HON  EyWELL-400 -X  BAS IC 
HON EYMELL-600 -BASIC 
HP2000C-BASIC 
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I1P2000F-BASIC 
HP 3000- BASIC 
IBM-S3~HOD-6-BASIC 
LEAS CO-BASIC 
HONEYWELL -1640-BASIC 

^fULTICOMP-BASIC>:  (UNIV  OF  MASS,  AhfflERST,  CDC-3600) 

WANG-3 300 -BASIC 

UHIVAC-llOO~UBAi;iC 

XDX -BASIC 

WANG-2200-BASIC 

Therefore,  following  the  recommended  translatability  rules,  a  user  sluiuld 
be  able  to  obtain  a  system  tnat  is  translatable  with  a  minimum  of  effort 
and  time  within  the  third  category  and  translatable  with  greater  difficulty 
and  expense  in  the  second  category.    It  should  be  noted  thnL  a  protiram 
usually  runs  at  a  slower  speed  on  a  small  machine  than  on  a  large  inaciilne. 

The  inforniation  provided  above  is  a  synopsis  of  extensive  ch;irts 
comparing  the  above  Jialocts.    These  charts  are  available  from  tho  ^uit hoi . 
All  information  was  obtained  frcm  manufacturers'  manuals  and  is  subject  to 
change.    It  can  clearly  be  seen  that  BASIC  translatability  is  a  faft  iiikI 
can  be  performed  easily  if  a  few  rules  are  followed. 
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The  purpose  of  this  set  of  tables  is  to  supplement  the  basic  report  on  Interdialert 
Tranglatabllitv  of  the  BASTn  Programming  Language  (Gerald  L.  Isaacs,  Technical  Bulletin  tvV,  II, 
The  American  College  Testing  Program)  and  to  give  a  quick,  clear,  concise,  updated  view  of  niauv  ^ 
of  the  BASIC  languages  af;  supported  on  different  computer  systems.    As  shown,  each  system  h;i^ 
irs  own  set  of  commands  and  its  own  set  of  capabilities*    Preceding  the  tables  is  a  listing 
of  the  conventions  used  in  the  tables,  a  summary  of  file  capabilities  of  the  various  systems 
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a^tzined  100%  accuracy  and  even  if  ve  did  that  accuracy  would  soon  decay  as  a  result  of  the  ^ 
continuing  fast  pace  of  improvement  now  evidencing  itself.    We  should  note  Specifically  that 
we  have  not  credited  various  dialects  with  features  that  are  "promised  for  delivery  in  the 
near  future"  or  even  those  which  we  are  told  exist  but  "are  not  yet  documented"*    For  this 
reason,  we  U7:ge  any  potential  user  to  check  with  the  relevant  manufacture  before  dismissing 
from  consideration  any  system  that  seems  attractive*    At  the  same  time,  we  urge  manufacturers 
to  supply  us  with  documentation  of  improvements  so  that  we  can  keep  our  charts  up  to  date. 
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The  following  convention;?  ;ire  used  in  the  tables. 


not  available 

num  or  n 

number 

var 

variable 

expression 

ill*  o 

a  vgume  n  t 

numlls  t 

number  list 

val  list 

value  list 

CO 

var  list 

variable  list 

O 

op 

operator 

str 

string 

par  am 

parameter 
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BURR0UCHS-B2500  none 

BURROUCI3S-B3500  none 

GENEitAL  AUTOMATION  BASIC-16  ADVANCED  none 

HOKEYUELL  316,  516,  and  716  none 

IBH/C?S  <Unlv  of  Iowa)  none 

HICRODATA  none. 

N'CR  CENTURY  100  none 

NCR  CENTtmY  200  no  files  as 

Q-DATA  BASIC-1  none 

UWICOMP-COMP  16,  COMP  18  none 

UNIVAC  1100  <Unlv  of  Maryland)  none 
Version  1-3 

WESXIKGHOUSE  BASIC  II  none 

WESTINGHOUSE  BASIC  III  none 


BURR0UGHS-B5500  sequential 

CDC  6000-KROtfOS  sequential 

CDC  6000-SCOFE  Bequenclal 

DEC-FDF8  (Edusystem  25  and  50)  sequential 

QE  255  Time  Sharing  Extended  BASIC  sequential 

HOKEYWEIX  1640  sequential 

IBH/CALL/360-OS  sequential 

IBM/ITF  sequential 

IBM  S3  MOD  6  sequential 

UCSIHB6700  sequeiktlal 

UNIVAC  1100-UBASIC  (Mankato  State)  sequent!;*! 

Version  2*0 

U*AN<;  3300  sequential 

WAKC  2200  sequential 


Capability 


BASTC  ^-BUSINESS  BASIC 

sequential 

and 

random 

access 

COM-SIIARE  BASIC 

sequential 

and 

random 

access 

COK-^  SHARE  NEW6ASIC 

sequential 

and 

random 

access 

DARTMOUTH 

sequential 

and 

random 

access 

DATA  GENERAL 

sequential 

and 

random 

access 

DEC-PDP  10 

sequential 

and 

random 

access 

DEC-PDP  11 

sequential 

and 

random 

access 

QE  MARX  I 

sequential 

and 

random  access 

QE  MARK  II 

sequential 

and 

random 

access 

GE  MARIC  III 

seq uentlal 

and 

random 

access 

HONEYWELL  200 

seq uentlal 

random 

access 

HONEYWELL  400 

and 

disc  arrays 

3  c  U  UV  1 1 1>  X  d  ^ 

random 

access 

HP2000B 

sequential 

and 

random 

access 

HP2000C 

sequential 

and 

random 

access 

HP2000E 

sequential 

and 

randon 

access 

HP2000F 

sequential 

and 

random 

access 

HP3000 

sequential 

and 

random  access 

LEASCO-RESPONSE  I 

sequential 

and 

random 

access 

MULTICOMP  BASICX 

sequential 

and 

random 

access 

UNIVAC  1100-UBASIC 

Version  3#2 

sequential 

and 

random  access 

XDS-BASIC 

sequential 

«nd 

random 

access 
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CDC  DOOU 

SCOPE 

I5^f 

CPs 
IWIV  OF  IOWA 

GE  MARK  U 
GEMARK  HI 

HP200DB 

IEP200DC 

ACCESS 

JL4  OlOUS 

14  DIGITS 

6-8  DIGITS 

6  DIGITS 

9  DIGITS 

6-7  DIGITS 

6-7  DIGITS 

AND 

AND 

ARRAY  ^AME** 

letter  or 
letter  nun 

lett^sr  or  C 

Issuer 

letter 

letur 

lett^sr 

letter 

ARRAV  JTA.ITIXG'* 

1 

0 

0 

0 

0 

1 

1 

Assia\' 

se«  FILE 

see  FILES 

ASSIQf  narai^t 
nuQp  var^  inask 

BACSSPACE 

BACKSPACE  tl  cicp 
^-VCKSpYVrES;  ej5> 

CALL 

CALL  '^ME*': 
arglp  argZ 
CAlI  by  ref 
call  by  vitltx 
argl 

GVLL  mxn, 
list 

CALL  routine 
n3!TC  not 
sza.^dard 

wm;  fUc  -mini, 
Mir-^p  ... 

CHAI^'  nar« 
CHXIN"  naw^ 

CHMS'  nanse 

OV^IU  n^fl^ep  ' 

ITF 

LEASOO 

POP  10 

POP  11 

UNIVAC  1100 
MIWKATO  STATE  CLG 

MULTICCMP 
OR 

RASICX 

XEROX 

ACCESS  i 

ACCESS 
num,  code 



/CCURACY** 

15  DIGITS 

7  DIGITS 

S  DIGITS 

15  DIGITS^ 
dctibl^e, 
precis  Ion 

8  DIGITS 

11  MGITS 

16  DIGITS 

AND 

AND 

AM) 

ANDCe)q>lp 

APPFDXDttTELY  EqUAL 

m  m. 

ARRAY  NAME** 

letter 

letter 

letter 

lettei  followed 

letter 

letter 

letter 

ARRAY  STABTJWG** 

1 

1 

0 

0 

D 

0 

1 

ASSICM 

AS5IQ4  Otm 
nun^  Vdip  lusKJ 

see  OPB^ 

ASSIOI  Tum  to 

BACKSPACE 

CALL 

CALL  name 
(pai^meter  list) 

CHAIN  none 
line  1 

CHAIN 

CHAIN  naro, 
lir.e  1 

OiAIN  naETp  n  or 
or 

atAIN:  nape,  n 
or 

aiALV:  'name,  n 

CHAIN  naix 
clears  storage 

passwond:  nm 

**Conuikdft  «Dd  client*  chit  etn  be  used* 


ERIC 


IBM 
CALL/3ffO-OS 

pDp  a/E 

200 

CDC  6000 
miSQS 
E^ir  2,0 

NCR 
CEKTURV 
200 

UCSD* 

BASIC 
B6700 

nP2000F 

Aa:ES$ 

ACCURACY** 

IS  DIGITS 

7  DIGITS 

10  DIGITS 

14  DIGITS 

7  DIGITS 

H  DIGITS 

6  to  7  DIGITS 

AND 

AND 

■■ 

APPK»CimT£LV  BqMW- 

ARIUV  KA>tE*^ 

letter  or  1 
or  9  or  ^ 

letter 

letter 

letter  or 
1  e  1 1  e  t  nu»^*jc  I 

letter 

letter 

letter 

AltRAV  SIMniKG** 

I 

0 

0 

1 

0 

I 

ASSIQJ 

A^blun  n^ue j 
ilun  f  name , 
irask 

BACKSPACE 

BACKSPACE  #exp 

CAIX 

CALL  name 
CALL  n;^me 
nuin 

nijct»ij5er  * 

- 1  -  -  - 

CHAIK  name 

CHAIN  ntae. 

CJRIK  name 
CHAIN  name 
num 

nUn-user  f 

*us;:vE«sir;  or  California^  saw  otbco,  burrouchc  b67O0 

O  **CA?=UAds  tod  «I««ent«  ctn  be  u»ed. 

ERIC 


10 


ACCESS 


AOCUBACif** 


AND 


APPROXiMAtELY  BqMAL 


ARRAY  WWE** 


AWUY  STARTING** 


ASSIGN 


au:KSPACE 


CALL 


CENTURY  100 
BASIC  1 


7  DIGITS 


letter 


BURAaUGH<; 
BSSOO 
BASIC 


11  DIGITS 


BUKKOUGKS 
B2S00 
BASIC 


a  DIGITS 


letter 


fitJRROUGHS 
BSSOO 
BASIC 


BASIC  FOUR 
BUSINESS  BASIC 


8  DIGITS 


1*  DIGITS 


letter 


letter 


**CbcTaJVls  and  elements  that  can  be  usrf. 


UNICOMP 
COMP  16  or 
COMP  IS  JASIC 


7  DIGITS 


letter 


VARIAN 
6Z0  or  V73 
BASIC 


^  7  DIGITS 


AND 


letter 


IP 


CALL  nanep 


ERLC 


IBM  53 

m>  6 

GE  ZSS 
TIME  SHARlKG 

OCM-SKUIH 
BAilC 

OOM-SHARE 
NEWEASIC 

h-ESTlKCHDUSE 

b;^ic  II 

VpXSTJWGHXSE 

BASIC  in 

GENERAL  ALTOIATICN 
A0yAN*aD 

a^xsic-16 

ACCESS 

15  DIGITS 

9  DIGITS 

6  DiaTS 

10  DIGITS 
or 

IS  DIGITS 

7-S  DIGITS 

7-8  DIGITS 

6  DIGITS 

AND 

A,ND 

AM) 

AM) 

APimCDttTBLV  EQUAL 

ARRAV  KflME** 

letter,  S,  i, 
or  * 

letter 

letter 

letter 

letter 

letter 

letter 

ARRAV  STARTUJG** 

1 

D 

D 

1 

0 

0 

1 

ALLOCATE 

sec  FILES 

see  OPSf 

sec  OPEH 



BACKSPACE: 

BACKSPACE  f  exp 



CALL 

CALL  fiazi&r  l)jie 
ntfnber  cperates 
like  QtAL'f 

CALL  Tiiffe  or 
CAU  FN  letter 
cr 

CALL  S  riairc 

CALL.  (lUUD, 
e>q?  1,  eiqp  Zf 
•  *  •) 

GALL  nme 
Ce>^  1,  exp  2, 

CHAIN 

see  CALL 

a*AlN  na:ne 
or 

CHAIN  nane, 
line  * 

SCRATCH 
PROCEED 
Provides  nanual 

LINK  '"Ituire!" 
LOAD  "Ijiarftj" 

^*C6!niumdis  and  elcmnts  th«t  ean  be  used^ 


UN1VAC  IlOO 
UBASIC 

irWEWELL  16^0 
X^IC 

HDNEYWELL^ 
316,  $16,  and 
716  BASIC 

HOKEmU,  400 
XBA5IC 

>  ICffEYKBa  600 
BASIC 

UNIVAC  HOD 
UNIV  OF  MUmjW 
RELEASE  V  1,5 

ACCESS 

ACCURACY** 

a  DIGITS 

»  DIGITS 

'^*  6  DICITS 

II  DIGITS 

'v*  8  DIGITS 

6  to  7  DIGITS 

8  DIGITS 

AND 

AND 

APPflDXlHATaY  BqpAL 

ARRAY  KWE** 

letter 

letter 

letter 

letter 

letter 

letter 

letter 

ARRAY  STARTING** 

0 

0 

0 

0 

1 

1 

0 

ASSIOI 

see  FILES 



BACKSPACE 

BACKSPACE  1  ntn 

BACKSPACE  1  mAn 
BACKSPACE  :  mrt 

CALL 

CALL  I^Ce^cp  1, 
exp  n) 

CALL  jmm 

CALLCnum,  exp  1, 

Fortran  or 
Asserfcler  (My 

CALL  n^mft 
or 

CALL  name 
options 

CALL  iui% 
or 

CALL  nam, 
password 

CAa  RINC 
(e^tp  1, 

DI^IK  name, 
nuci  or 
□IAIN:  or 
CKAIK*  or 
CHAIN;* 

01AIN  RUN: 
najpti  options 
RUN  KAY  be  any 
RUN  conrjnd 

CHMN  nanie,  nun 
QtAIN  name 
Q'JVIN  nane, 
password, 
nisn 

CHAIN 

IF 


'^onnmds  and  elements  that  can  !;e  used. 


MICRDQATA 
BASIC 


a\sic-i 


HP3000 


KANG  3300 


ELEORIC 
MARK  1 


KANC  2200 


JVDCESS 


ERIC 


ACCURAOf** 


AND 


APPROXIMATELY  SJUAt 


ARRAY  f«ME** 


ARRAY  STAKTING** 


ASSIGN 


BACKSPACE 


CALL 


QtAIN 


9  DIGITS 


letter 


6-7  DIGITS 


letter 


6-7  DIGITS 
11-12  DQUBI^ 

PREC  SION 


A.^ 


letter  or 
letter  digit 


e^p  varp  ms^ 


CALL  none 


CHAIN*  namCp  e^ 


8  DIGITS 


AND  Cejq>  1, 
e^  n) 


letter 


CmiS  R  iiame 
CHALV  ntm 
CmiK  R  ntBn 


9  DIGITS 


letter 


BftCKSPACe  f  exp 


CALL  n£u:)e 


CHAIN  nanie 
CHAIS  n^icve,  nijn 


13  DIGITS 


letter 


SELOCT  options 


GDSUB*  Ttm 
(var  Ip 
var  n) 


LOAD  tupie, 


^O^nnands  and  elenentf  that  ctn  bo  usc<3. 


CO 


CDC  66C0 

IB^ 
CPS 
tWrV  OF  ICMK 

i!ATA  d^iStAL 

HAM  II 
GE  MMtK  in 

JP2000B 

IIP2000C 

CHAHGE 

OttNGB  nunlist 
X.O  sirir^  BIT 
aWSGE  string 
to  nunlist 

CHANCE  nuaUst 
to  striitg 
CHWWE  String 
to  nxEilist 



,   

CLOSE 

- 

CLOSE  FlLEexp 

OC»M>N(FrLE) 

CCM  list 

CCM  list 

CONCA.TEKATION 

+ 

t 

DMA**  " 

DATA  val  list 

DATA  ejtp,  ... 

DATA  val  list 

DfTX  val  list 

DAT^  exp,  ... 

DATA  val  list 

DATA  val  list 

DATA  FILE 

- 

DEF** 

DEF  FIWC 
(var)  -  e)(p 
cne  line^ 

DATA  HJSC 
(var  list)  » 

one  line 

DEF  FUSC 
(var  list) 
one^  or  roult 

DEF  FUNC 
one  line 

DEF  FIWC 
(var  list) 

OJW  or  EMlt 

DEF  FUXC 
(var)  -  eyp 
ono  line 

DEF  FUKC 
(var)  ■  ejqp 
one  line 

DIM  *  (virtual  stor) 





DIMENSION** 

DIM  xiatA 
<je fault:  ID 

DIK  mix 

DIM  nanc 
(OiiD^nsicns), ..  * 
defttult  10 

DIM  nane 

(dino^nsloiis) ,  .. . 
^^fault  10 

DK  narc 

(<!3)SCflSi«Ti£),  ... 

default  10 

DIM  iMlt» 
(dir<*;iSlons) J 
<3e  fault  10 

DIH  nan^ 

(din^nsions) ,  ... 
default  10 

EvD 

END 

END 

ENDFTLE 

feo  IF  END  ' 

sw  IF  EM) 

s«o  IF  DO) 

**C^3ain4i  iQd  elcn^oca  ttiit  cm  bo  u«ed^ 


ITF 

LEASOO 

POP  10 

PDF  11 

UKIVAC  1100 
UEASJC  VEftSlOftf  2,0 
ST,\TE  CLG 

HULTICCHP 

OR 

UN'IV  MJ^SS 
EASICX 

XEROX 



OONV 

CHWCc  string  tc 
CHANGE  iiura  to 

StTiiTg 

QWKCE  ntun  to 
string 

QfAKGE  string  tc 
nL  J) 

CIWJGE  nuralist  t<? 
scring 

aiAN(£  String  to 
nuRil  ist 

CKAXGE  nuTJist  to 
stritig 

aVWGE  string  to 
ntsnlist 

CHAKGE  string  to 
nin 

QiWGE  nuh  to 
string 

CLOSE 

CIjOSE  FILE 
'damc' t  *  *  * 

Close  list  of 

CLOSE  nurif 

CL3SH  M ,  nanie 

CLOSE:  ntjci 

CWON(FILE) 

CCMDS  n 

CCtOTEKWION 

■  + 

+ 

CAT$(strl,  str2) 

+ 

DATA** 

DATA  val  list 

DATA  val  list 

UTA  val  list 

EtATA  val  list 

DATA  val  list 

MIA  val  list 
DAlU  val:Tep 

QATA  val  list 

lUTA  FILE 

DATAFILE  * 
M)  nsone 



DEF** 

DBF  mc 
one  liiM 

DEF  FUNC 

(var  list)*exp 

OTJ^  line 

DEF  FWC 
(var  list)  one 
or  nult  line 

DEF  FUWC 
(var  list) 

DEF  RJWC 
(var  list) 
one  or  ciult 

DEF  FUJC 
(var  list),  eatp 
line 

DEF  FUfiC 
(var  list)-e3q) 
one  line 

DIM  *  Cvlrtt3»l  5tor) 

DII^  i  n 

DIM  name 
defatjtlt  10 

DIM  nane 

(dimensions)  >  , , . 
default  10 

DIM  nunc 
(diniCTisions) , 
defatjtlt  10 

DIM  nai» 

{di  y^ns  ions)  >  , , , 
default  10 

DIM  najic 
(diricnsions), 
default  10 

DIM  name 
(dimensions), 
defatjtlt  10 

DLM  naw 

(^tncnslons)  t  -  -  - 
ddfatjtlt  10 

END 

END 

END 

END 

ET^DFILE 

see  OH 

EM3FILE  n 

&DFILE;  e^* 
line  i 

««CDiiBaad5  tnd  cl«ffcnt«  Ui«r  c«n  be  u^ed. 


IBM 

CALL/360-OS 


PDP  8/E 


HONEYWELL 
200 


CDC  6000 
^WSIC  2*0 


zoo 


UCSD* 

BASIC 
B6700 


HP2000I^ 


C»\NC£ 


ERIC 


CLOSE 


OCKMX^(SPOn^GE) 


COtCATEWTlCW 


DATA  FILE 


DEF* 


DIM  f  (virtual  stor) 


DIMEKSION^ 


END* 


Close  expij 


DATA  val  list 


DEF  FUNC 
Cv4r)*exp 
on*  line 


Dili  nane 
(dimensions) p 

default  10 


End  comment 


DATA  val  list 
no  strings 


DEF  FUNC 
Cvarl,*  *  * )»exp 
one  line 


DIH  name 
(dimensions) , 


END 


U^r'EPSm  Cf  CALIFORNIA,  SA2i  &iECOp  BURHOyCKS 
'*Co=:aiQd«  «n<l  elcacnc*  Chac  etn  be  used^ 


DATA  val  list 


DEF  FUNC 
Cvarl,...)-^'fp 
ont  lint 


DATA  val  list 


DEF  FUNC 
(var)-exp 
one  line 


DIM  name       j     DIM  name 
(dimensions) p  I  (dinensions) t 

default  10         default  10 


£!vD 


stt  IP  EM) 


see  ^iODATA 


DATA  val  list 


DIM  name 
(dimensions) t 


EN"D 


DATA  val  list 


DEF  FU»C 
fvarl , . *  4 )"^xp 
one  line 


DIH  name 
(dimensions) , 

default  10 


!^ee  IF  Et:D 


COM  varl,. 


DATA  val  list 


DEF  FUNC 
(var)-exp 
one  line 


DIH  nv^c 
(dimens  ions) , 

default  10 


ETiD 


see  IF  END 


NCR 
CENTURY  100 
BASIC  1 

BURROUGHS 
BSSDO 
BASIC 

BURROUGKS 
B2S00 
BASIC 

BURROUGHS 
B3S0O 
BASIC 

BASIC  FOUR 
BUSINESS  BASIC 

UNICOMF 
COMP  16  or 
COMP  1$  BASIC 

VARIAN 
620  or  V73 
BASIC 

CLOSE 

CLOSE  Cnum) 

COMOKCFILE) 

co^3CATE^w^I0N 

DATA  vsl  list 
no  strings 

DATA  val  list 

OATA  val  list 
no  strings 

DATA  val  list 
no  strings 

DATA  val  list 

DATA  val  list 

DATA  FILE 

DEF** 

DEF  FUNC 
one  line 

DEF  FUNC 
one  line 

DEF  FlJ^C 
(var)  -  ej:p 
one  line 

DEF  F^J^rC 

(var)  -  exp 
one  line 

DEF  FUNC 
(varl,  * .  .  )**^^P 
one  line 

DEF  FUNC 

L  V  •  1  J 

one  line 

DIM  t  (virtual  stor) 

DIMENSION** 

DIM  var 
(dimensions) ^ 

default  10 

DIH  var 
(dimensions) ^ 

default  10 

DIM  var 
(dil&ensions), 

default  10 

DIM  var 
Cdin'ensions)^ 

default  10 

DIM  var 
Cdinens  ions) , 

DIM  var 
(dimensions) ^ 

default  10 

DIM  var 
Cdinensions)^ 

default  30 

eo)** 

END 

END 

EKD 

END 

END 

END 

E?iDFILE 

S^e  READ  t 
see  WRITE  r 

**C<]nvnands  a^d  eleinents  that  can  be  uscd^ 


^  1 


lEM  S3 
EASIC 


GE  255 
BASIC 


GCM-StARE 

BAsrc 


KEWE^SIC 


t^-ESHNacUSE 
BASIC  II 


vs'Esns-aousE 
a.v>ic  III 


CENERAi 
RA<;jC-16 


2E 


CHAi*GE 


CHAKGE  str  TO 
array  rar 
CHWGE  array  TO 
string  v^r 


CHA.VGE 
is  a 
string  function 


CLOSE 


(X06E  nun  or 
str  var,  * . . 


CLOSE  imiT 
CLOSE  otnpt/r 


OjOSO  exp 


OM^  (FILE) 


OCrtDK  (STORAGE) 


OOSCATEKATIOM 


DATA** 


DATA  FILE 


DEF*" 


DIM  *  (virtual  stor) 


LET  STR  ■ 
(str  var^  ntfti^ 
ttum)  ■  Str  var 


DIVTA  val  1,  val 
^,  ...  val  n 


DEF  FUNC(var)- 
single  lixvt 


DI^^E^SIO^*•* 


END** 


ENDFILE 


DIM  var(nwu,  nxinj) 
default  10 


END  cocmcnt  or 


GATA  val  1,  val 
2,  ...  val  n 


DBF  nJNC(^'ar)- 
exp 
single  line 


DIM  var(nm, 
mcflt) ,  ... 
<!efdiat  IQ 


see  IF  E^D  * 


DATA  val  I,  val 
2,  . . .  val  n 
no  strings 


DEF  IWC(var)- 
e^p 
single  linn 


DISC  ^-arCTUjn) , 
. . .  DISC  var 
Cntmt,  nun)... 


DIM  'i-ardui:;, 
nuri),  ... 

<!c  fault  10 


END 


DATA  val  1, 
val  n 


DEF  FUNC(var  1, 
...)  -  exp  or 
DEF  FUKC(var  1, 

nultiple  11 


DLM  v.irCeXp, 
...),  ... 
default  10 


END 


lin^  ' 


DATA  val  1,  val 
2f  ...  val  n 
no  strings 


DEF  FlNCCvar  1, 
...3  -  exp 
single  luie 


DI.^var(n,  ...), 
aefiait  1-10 


END 


DATA  val  I,  v^l 
2,  ...  val  n 


DEF  mcC(vsr  1, 
...)  "  exp 
Single  line 


DIX  var(n,  ...), 
default  n*10 


COM  var. 


DMA  val  1,  val 
2p  ...  val  n 


DEF  FUHCCvar  1, 
...)  -  exp 
single  line 


DIM  var(n,  .. 


END 


**Conr*:mds  .ind  elements  that  can  be  used. 


ERIC 


CO 


20 


tfflVAC  1100 
UBAStC 


HONHVNEIX  1640 
XBASIC 


HOKEVmL 
316,  S16,  and 
716  tyvSIC 


HONCnmU.  400 
XBASIC 


BASIC 


HP2000E 


tWiVAC  1100 
WIV  OF  mRYLAM) 
RELEASE  V  1,3 


2F 


CHAMGE  string 
TO  var 

O^AMSE  var  TO 
string  


CH^v^aGE  string 
TO  var 

Q[ANGE  var  TO 
String  


OttNGE  string 
TO  v:ir 

aiVVCE  var  TO 
string  


av^ijB  string 
TO  var 

a!A^'CE  var  TO 
string  


CLOSE 


CLOSE  exp  1, 


CLOSE;  n*w: 


see  SCRATCH 


CaHON  (FILE) 


OOrrv  (STORAGE) 


CONCATBttTlCW 


CATSCstr  I, 
str  2) 


EtATA  val  1, 
vnl  n 


EtATA  val  1, 
val  ^ 


DATA  val  It 

val  n 

no  strinfis 


OATA  val  1, 
val  n 


tlATA  val  1, 
val  n 


OCJJ  var  Ij 
var  n 


mTA  val  I, 
val  ^ 


OATA  val  1, 
val  n 


aATA  FILE 


OEF  FUNCCvar), 
. var  n)^trp 
DEF  FUlCCvar  1, 
vjtr 


KF  FlWCfvar)- 
single  line 


DEF  FUNCfvor)- 

e)q> 

single  line 


DEF  nmc(var)- 

ejtp 

single  line 


DEF  HJMCfvar)- 
exp  or 

DEF  FUNCCvar)^ 
„ , *  var 


DEF  FUNCCvar)- 
exp 

single  lijM* 


DEF  HM:Cvar  1, 
var  n)»exp 

DEF  FUNCCvar  1, 
var  n) 


DIM  *  Cvirtual  stor) 


DISC  varCnurtp 
nusO:  none 


DI?iE>aSlCfi** 


DIM  varCnun, 
mm  1,  n'jn  2p 
mm  3) p     r  or 
ARRAYS  to  sped 
default 

fault  10 


DIM  varCr,wi, 
mm),  „. 

default  10 


DIM  varCniOTj 
default  10 


DIM  varCniaap 

F  <  <)  ,      +  .  . 

default  10 


DIM  var  Cms^* 
default  10 


DIM  var  (man, 
nua) ,  ... 

default  10 


DIM  varCnum, 
default  10 


END 


EM) 


EXT) 


END 


END 


ENDFILE 


exp  GOTO  line 


see  IF  END 


see  IF  e^) 


see  IF  EW) 


see  IF  END 


**Coirmndls  and  elements  thct  can  be  used. 


ERIC 


CHANGE 


CLOSE 


GDMW  (FILE) 


CCJMCJJ  (STORAGE) 


MICRODATA 
BASIC 


<l-nATA 
&A5IC-1 


HP3000 


CONVERT  nun 
TO  string; 
OONVEBT  string 
TO  v.ir 


OCM  list 


wc^^G  3300 


FILEEND  ^  num 


OCW  var  1* 


EUmiC 
MARK  I 


2200 


. 


COM  v^r  It 


1^ 


COCATEWION 


DATA** 


a\TA  FILE 


OEF* 


DIM  i  (virtual  5tor) 


END** 


S^FILE 


DMA  val  1, 
v»l  2,  val 

no  strings 


D£F  FUhC  (var  1, 
v«r  2, 

var  n)  ■  exp 
single  line 


DIM  var  (dim, 
dcf^lt  10 


mTA  val  1, 
val  2t  ...  val 
no  strings 


DEF  FUJJC  (var) 
=  exp 

single  line 


DIM  var  (diro, 
din)^  ... 
default  10 


E^^ 


DATA  val  1^ 
val  n 


DEF  mJZ  (var  It 
. . .  var  n)  =  exp 
DEF  Fl^JC  (var  1^ 
...  var  n) 


QATA  vol  It 
val  n 


DEF  FUJC  (var) 
sinj^l<>  line 


DIM  var  (dim, 
din),  ... 
deCault  10 


ENT) 


see  IF  E?© 


DIM  var  (diet, 
dirdt  ... 


sec  IF  EM? 


DATA  val  It 
val  n 


DEF  HJfC 
-  exp 

single  line 


(var) 


DDI  var  (dim, 
din)^  ... 
defatdt  10 


DATA  val  1, 
val  n 


DEF  fun: 

"»  <>xp 

single  line 


(var) 


DIM  var  C<iiri, 
dir*),  ... 


END 


ERLC 


**Con»nands  and  elerwnts  that  can  be'  us<^d. 


BASIC  2.D 
COC  66D0 
SCOPE 

CPS 
IMIV  OF  lOKA 

QrvTA  GS^ERAL 

GE  MUUC  II 
GE  MARK  III 

tiP20CI0B 

IIP2000C 

ENTER  var»  expi... 

OKTEft  f  v^T^ 

exp»  ... 

BjTER  var,  exp^*. . 

■ 

M 

■ 

"  or 

* 

■ 



EXOiAKCE 

EXCLUSIVE  OR 

EXPONIOTlATiaV** 

t  or  ** 

** 

f  or  ** 

t 

FIHLD 

see  USING 

£<sc  ustrc 

see  U3IXG 

FitH 

FILE  #  cxp: 

see  OPEN* 

FILE  '  ntm,  nasc 
FILE;  ttwt,  ttssTC 

see  FILES 

see  FILES 

FIUE  KAMEWOC 

8  char 

6  characters  not 
including  extensicnj 

,  ^  char 

^  char 

FILES 

! 

see  FIIE 

see  OPEN 

Files  narae, 

FILES  na^e 

FII£S  naiDe 

F^^END 

FOR  ** 

,  POR  var  ■  expl 
;  TO  ejcp2  STEP 

FOR  var  »  expl 
TO  cxpZ  STH> 

KIR  var  ■  e!«pl 
TO  eJCp2  SEEP 

FOR  var  ■  expl 
70  ejcp2  STEP 

FOR  var  ■  expt 
TO  exp2  STEP 
exp5 

FOR  var  ^»  exp: 
TO  exp2  STEP 
expJ 

FOR  var  ■  expl 
TO  exp2  STEP 
exp5 

GHPTR 

LGii'Cexp) 

**CotLA«n<ls  iTid  eleat^nts  th«t  c«n  be  tj&ed. 


lEK 

LEASOO 

POP  10 

FDP  11 

iSSIVflC  1100 

uHAsic  vmiav  2,0 

MWIKATO  STATE  CLG 

MULTIOOMP 

OR 

iwn^  ^tts^ 

BASICX 

XEROX 

■ 

■ 

m 

BQU(cxpl,  cxp2) 

■ 

BQUrVAL03CE 

EQU(expl,  ejq)2) 

EXaiAKGE 

FIIJ;  exp 

E)faiAg^E  vl ,  v2 
Vl  '  »  v2 

OPEN  na.-nc  to: 
str,  PRIST,  OM 

EXCUUSIVE  OR 

GOR 

XDftCexpI,  ejq)2) 



EXPOeOOTIATlON  ** 

* 

*  or  A* 

t 

t  or  ** 

t 

f  or  ** 

FIELD 

see  IMACH 

£e«  PEtlNT  USING 

S«  PRINT  USING 

sec  PRI^T  USIKG 

FIELD  (speclp 
spcc2. . *spec  n 

see  PRI^T  USIKG 

PILE 

FILE  iH,  str... 
PILE!  N\  str... 

see  OPDJ 

sec  OPEN' 

see  ASSlQt' 

sec  OPEN' 

PILE  KAME  MAX 

6  char 

12  char 

7  char 

PIIXS 

PILES  narocp  ,^ , 

FILES  nanx^r  . 

S«  OPEK* 

sec  OPDS 

see  ASSIO: 

see  OPEN 

RiENG) 

FNEND 

FKE^D 

FN-ENT) 



FOR  var  *  ejq>I 
TO  ejcp2  STEP 

FOR  var  ■  ejq)l 
TO  cjqpa  STHP 

TO  tjcpz  snp 

FOR  var  ■  e>pl 
TO  e^p3  BY  exp3 

FOR  vnr  "  ejcpl 
:o  exr2  STEP 

FOR  var  "  expl 
TO  cxpZ  STEP 
exp3 

FOR  var  ■  e^pl 
TO  ejcp2  STEP 
ejq)3 

FOR  var  '  expl 
TO  exp:  STEP 
expj 

GETPTR 

GETPTK  ntm,  varlp 

1 

var2 

ERIC 


EXCtfv\GE 


EXOUSIVEOR 


FIELD 


FILE 


FILE  tiAME  5V0C 


FILES 


FNEiD 


CETPTO 


CALL/560-OS 


*  or  nn 


see  Pftitrr 


FILE  narae^num 
not  Collect 


8  characters 


see  FILE 


FOR  var-exol 
70  C3cp2  ST3P 


PDP  e/E 


FOR  vaT»expl 
TO  cxp:  STEP 


HONEYWELL 
200 


CDC  6000 
Sf^SlC  2.0 


*  or  A 


FMT  or 
see  PRINT 
USING 


see  FILES 


letter  or 
letter  nun! 


FILES  namcl , . - 


TO  cxp2  STEP 


or  ** 


7  characters 


7^?.  vav^eypl 


NCR 
CErCTURY 

:oo 


'CR  vat'expl 
TO  cxp2 
STEP  ^^cp3 


**Co=3d"«d3  and  elements  that  t^n  be  uscJ* 


ucstf 

BASIC 
n6700 


t  or  *** 


see  FILES 


i7  charaicters 


FILES  naraelp. - 


FOF  var'cxpl 
TO  exp  2 
STEP  expS 


nPZOOOr 


ENTER  ^  varl, 
EKTER  varl,..H 


see  IMAGE 


see  FILES 


6  characters 


FILES  Dame, *  * . 


FOR  var*expl 
TO  cxp  2 
STEP  exp3 


3C 


26 


NCR 
CENTURY  XOO 
BASIC  1 


BURROUGHS 
BSSOO 
BASIC 


BURROUGHS 
B2S00 
BASIC, 


BURROUGHS , 
B3S00 
BASIC 


BASIC  FOUR 
BUSI7JHSS  BASIC 


UNICOMP 
COMP  X6  OT  COMP 
IS  BASIC 


VARIA?J 
620  OT  V73 
BASIC 


3D 


or  + 


EQUrVALEJCE 


EXCLUSIVE  OR 


ECPONTNTUTION* 


integCT  <  ■  9 


FIELD 


see  IMAGB 


FILE 


see  FILES 


see  OPEN 


FILE  ^AM£  MAX 


6  chaTACters 


6  characters 


6  chaTacteTS 
progTan  files 
only 


FlliES 


see  0?2K 


FSEND 


ICR* 


FOR  var"e;q>  I 
TO  ejqp  2 
STEP  eyp  I 


FOR  var"c;q>  1 
TO  exp  2 
STEP  exp  3 


FOR  var"e>p  1 
TO  C7P  2 
STEP  exp  3 


var"e^  1 

TO  e:xp  2 
STBP  c>p  3 


FOR  var-exp  I 

TO  c;cp  2 
STHP  exp  3 


FOR  var-eicp  1 
TO  exp  2 
STEP  cxp  3 


FOR  var*e;q>  I 
TD  e>p  2 
STB?  eyep  3 


GETFTR 


KEY 


^*Cot?Kuids  tmd  eleinents  that  can  be  used- 


1 

1 

2 

1) 

t; 

m 

[;; 

FIELD 

i 

EXCLUSIVE  OR 

i  1 

1 

§  1 

8 

* 

— 

1?? 

Co 

o 

B" 

s 
1! 

5 

1 

o 

* 
* 

■J 

<* 

!! 

<* 

m 

1 

i 

»-*  5:  tJ 

5  m1 

tA 

r> 
o 

H 

S3 

*» 
<* 

i 

• 

3 
1 

tA 

o 

o 

S- 

tit 

a 

O 

* 
* 

i 

P  i 

it 

"  S 

-  - 

1- 

• 

u 

*^ 

»t 
1  * 

1 

t*t 

m 


430 


28 


UNIV^  1100 

HQKEYKELL  1640 

H^lNEmELL 
316,  SlS^snd 
716  &ASIC 

I1QKEWELL  400 

WNB^^fELl  600 

HP2(K)0E 

UXIVAC  1100 
UNIV  OF  >ttRVLA?a> 
ftEI£ASE  V 

exp  2)  or  " 

EQ  or  * 

T¥l^  f^vrt  1 

ejq>  2) 

EXOMGE 

ECatAXGE  var  1, 
var  2  or  var  1 
var  2 

EXCUUtJiVE  Oft 

XOR  (exp  1, 
exp  2) 

*  or  A  or  ** 

f  or  ** 

* 

f  or  ** 

* 

f  or  ** 

FIELD 

se^,  PRIKT  USING 

BfT 

5pv  W 1 11  &a  1 1  uTl5 

see  IMAGE 

see  IMAGE 

FILE 

se«  OPEN 

see  FILES 

FILE  f  mw, 
narie 

FILE  ff  nt!R,  name, 

password 

FILB*  nun,  nos^i 

password 

see  FILES 

FILE  NAME  MAX 

12  characters 

6  characters 

6  characters 

12  characters 

6  characters 

FILES 

see  OPEN 

FILES  name  t, 
. .  f  naiTo  n 

FILES  nanc  1; 
« « «  name  n 

FILES  na*TC* 
password; 
or 

FILES  options 

FILES  nare, 

FiCEND 

FND© 

FOR** 

FOR  var  ■  exp  1 

10  ej(p  2 
STEP  <*xp  3 

POR  var  "  exp  1 
10  ^xp  2 
STEP  exp  S  or 
FOa  var  -  exp  1* 
oxi>  2  S 

rMtt  var     exp  x 
?D  exp  2 
STEP  exp  3  or 
FCR  var     exp  1, 
e.xp  2f  <*xp  3 

tvK  var  ■  exp  x 
TO  exp  2 
STEP  exp  S 

FOR  Var  ■  exp  1 

10  ei<p  2 
STEP  3 

rUR  Var  "  exp  x 
10  exp  2 
STEP  exp  S 

FOR  V3r  ■  exp  1 
10  exp  2 
STEP  (?xp  5 
FOR  var  *  exp  1 
10  <?xp  2 
exp  3 

GETPTR 

**Co^'tfi<ls  an<)  elements  that  can  Im>  tcsc4. 


RASIC 

Q-DATA 
BASIC- 1 

1      WA!^;  330*> 

1 

t 

I 

!  1 

1  ! 

mm- 

|2^TER  J  var 
ENTER  nCT  I , 

! 

1  ; 

r 

p  —I 

\ 

j 

; 

• 

I 

1 

■ 

1  1 

t 

i 

i 

E 

t  i 

!  

! 

i 
1 

I  1 

EXCUJSn'E  OR 

! 

! 

  1   

1 

! 

""  " 

' 

! 

'      .  ! 
"  or  '  : 

!  :  

1 

•      i     •  : 

 \  \  ' 

f  j 
see  PRINT  USING  j  s^t?  U-'J^ 
t 

sec  ■ 

 i 

1 

i  ! 
\      ■      ■  ! 
 ■  1 

i  ! 

see  FILES          *  see  FTt^S 

!  i 

i 

sec  FILE5  j 

see  ASSTON 

i 

 — i 

!  : 

FILE  .W(E  iJAX                         ■    i   

t                              i  1 

depor*ds  or, 
installation 

I 

e  chiraciersi 

ch^ir3<;t<rrs      i  B  characters  j 

FILES  *  nun 
FEIXS  nxx 

"  1 

FILES  narc  i;     (see-  ASSIGN'          '  ♦ 
2;             \                         [  : 

* 

— 1 

* 

J 

  1 

1 

*  * 
FOR**                                    ;  ^OJt  v.*r  ^            '       var  •  eip  j[ 

exr  ^       exp  2    }  TO  *jq)  2 

Iv>;^  var  ■  cjp  l  J 
T^^  exp  :  1 

POU  v*^r  I  exp  1 
e^  J 

FOR  var  *  ejcp  J  ! 
TO  cxp  :  * 
?TrP  e*^*  3  i 

! 

v*ar  -  c>p  1  i 
'TO  ex?  2  ; 
STE?  ejq?  3  1 

t 

 ^ 

CO 


GHTPTP 


**Co!nrands  and  eleoents  that  can  w  o^.-s, 


J! 


BASIC  Z.O 
CDC  6600 

SCOPE 

I»S 

CPS 
UKIV  OP  IC«A 

EtATA  GIJ:ERi\L 

MARK  II 
GE  fl\RK  ITT 

HP2000B 

I1P2000C 

GOSUB** 

ODSUB  Itiut  t 

ODSUB  line  ' 

OOSLTB  line  I 

OOf^JB  line  ' 

COSdB  line  f 

OOSUB  line  t 

OOSUB  line  ' 

ODSUB  OP 

(coasted) 

seo  ON 

5t-e  0*i 

OOSUB  e;^  OP 
nlj  n2> 

GDSUS  exp  OF 
nl,  nZ, 

GOTO** 

0(710  line  * 

GOTO  nun 
GO  TO  nua 

03  TO  line  t 

00  ID  line  i 

GO  TO  num 

GOTO  line  r 
GO  TO  line  ' 

00  TO  line  * 
OOTO  exp  line  ' 

GOflO  OF 

(cocputed) 

SttC  ON 

GOTO  exp  OF 
nl,  n2^ 

OOTO  exp  OF 
nl,  fit,  ... 

GREATER** 

> 

> 

> 

> 

>  or  .GT^ 

> 

> 

CHEATER  BQUAt** 

>  ■  or  ■  > 

>     or  "  > 

>  •  or  ■  > 

>  * 

>  -  or  .GE* 

>  m 

H3LD 

IF** 

IP  en>l  OP  cxp2 
THEN  liwr  t 

IP  e:cpl  OP 
HEN  line  f 

IF  ejtpl  op  exp2 
TtW  line  *  or 
IP  expl  op  expZ 
GOTO  line  ' 

IF  expl  op  exp2 
THEN  line  t 
IP  expl  OP  ^^2 
GDIO  line  f 
JP  expl  op  e:cp2 
GCSU3  line  ' 
IP  expl  op  exp2 
TMBi  statement 

IF  e:cpl  op  e:cp2 
THEN  line  t 
IF  eMjl  OP 
COro  line  ' 

IP  e)ml  op  exp2 
THSJ  line  ' 

IP  expl  op  e:cpZ 
THEN  line  * 

IP  S^D  f 

IP  END  * 

eyp  THRf  line  f 

IP  EOP  (num) 
THEK  line  i 

IP  END  t  esq)  TOEN 
line  1 

IP  SfD  t  e:cp: 
THEN  line  1 

IP  EtOJ  1  ext^ 
THEN  line  i 

IP  BtD  t  e:cp 
Tie4  line  t 

IP  MDRE  ' 

IF  MDRE  t 

IP  MOEtE  t  typ 
l\m  line  ' 
IP  MORE  t  e:cp: 
IIBC  line  # 

ftftCocsand^        elements  ctut  can  be  used. 


i2 


I&1 

ITF 

LEA503 

VfJfP  10 

POP  11 

wiVAC  noo 

MAKKATO  STATE  CLG 

MULTiaM* 

iwrv  M\ss 

QAStOC 

XDV)X 

009JE  line  r 

OOSUB  line  < 

ODSUB  line  * 

a  SUB  line  f 

OOSUB  line  f 

GOSUB  line  ^ 

ODSUB  line  f 

OOSUB  OF 

GPSUB  era 
o;  line 

GOflD** 

GOTO  line  f 

GOrrO  line  I 

GO  TO  line  f 

GOTO  lin^  ' 

GO  TO  line  » 
CO  TO  *  num 

GO  10  Hne  « 
0010  Qyp,  line  * 

GOTO  line  » 

aSTO  OF 

(ccqputed) 

GOTO  exp 

of  I  inc  1  p  ,  ^  ^ 

GREATER** 

> 

> 

> 

> 

GTOCexplp  ejcpZ} 

> 

> 

GREATER  EQUAL** 

>  ■ 

>  ■ 

>  ■ 

>  ■ 

GEQCexplp  expZ) 

>  « 

>  ■  or  ■  > 

tDLD 

IF** 

IF  expl  op  expZ 
IHBT  line  f 
IF  expl  cp  expZ 
GOTO  line  * 

IF  ejml  op  exp2 
THEN  line  f 

IF  erol  op  en>2 
THE^  line  I 
IF  expl  op  e)ip2 
TTEN  line  1 

IF  expl  op  expZ 
71&  statenient 
IF  expl  oo  expZ 
TIO  line  ' 

expl  op  expZ 
OJIO  line  ' 

IF  'expl  op  e)ip2 
THeJ  line  * 
IF  tvfl  op  exp^ 
IHEN  statement 
IF  e^  cp  expZ 
GOTO  line  * 
IF  expl  op  e)5>Z 
GOaJB  line  ' 

IF  expl  op  e)ip2 
TTO  lijie  1  or 
IF  expl  op  escpZ 
COlO  line  ' 

IF  expl  op  expZ 
WDi  line  I 

IF  HMD  1 

IF       1  exp 
TllSJ  line  1 

IF  END  Ivar  TTEJ 
line  f 

IF  ©ID:  var  "IH©* 
line  ' 

see  ON 

IF^c}^^E  i 

**CoouQdft  and  el«£ents  th«t  e«n  be  used* 


ERIC 


IBM 

CALL/ 360 -OS 

HONEYWELL 
200 

CDC  6000 
KROWOS 
RASIC  2*0 

NCR 
CENTURY 

UCSD* 
BASIC 

dQ/UU 

HP2000F 

GOSUB** 

GOSUB  line" 

GOSUB  linei 

GOSUB  line* 

G:>SUB  linel 

GOSUB  linef 

GOSUB  linel 
GO  SUB  line* 

GOSUB  line  * 

COSJB  OF 

(computed) 

GOSUBCline*, 
4  Une')cxp 

GOSUB  exp  OF 
line' , *  *  * 

GOTO** 

GOTO  linef 

GOTO  line* 

GO  10  ILncf 

GO  TO  linel 

GO  To  lin<!* 
GOTO  line* 

GO  TO  linel 
GOTO  linel 

GO  TO  linel 
GOTO  line* 

GOTO  OF 

(computed) 

GOTO  linel,444 
line^  OX 

cxp 

GO  TO(line- , 
.  4  4  line* )  exp 
or  see  OK 

GOTO  exp  OF 
line*  j4  4  4 

GREATIER*^ 

} 

CREATER  EQUAL** 

* ■  or 

J-  or  »* 

»-  or  »» 

»»  or  »* 

KDLD 

IF** 

IF  expl 
op  exp2 
TH£M  ILnef 

IF  expl 
op  exp2 
GOTO  line  * 

IF  expl 
Op  exp2 
Tdnh  linef 

I?  expl 
op  exp2 
GOTO  line  * 

IF  expl 
op  exp2 
THEN  iinei 

IF  expl 
op  exp 2 
GO  TO  liiiis  1 

IF  expl 
op  exp2 
THEN  line  * 

IF  expl 
op  exp2 
TH£n  line* 

IF  expl 
op  exp2 
GO  TO  line  * 

IF  expl 
op  exp2 
THEN  line* 

IF  expl 
op  exp2 
THBN  line* 

IF  E©  1 

IF  £M)  1  name 
CO  TO  linel 

IF  END  *  name 
THEN  line* 

see  N'ODATA 

IF  END  i  exp 

IF  END  *  exp 

1  i  n»# 

Xr  MJKc  • 

*VSlVEaSITY  OP  CALIFORNIA,  SAM  DIBCO,  BURROUGHS  B6700 
^^Cocsiaads  and  eleaeitcs  chac  caq  be  utted. 


CO 
C5 


NCR 
CFMTURY  100 
BASIC  1 

BURROUGHS 
BASIC 

BURROUGHS 
B2S00 
BASK 

BURROUGHS 
*  B3S00 
BASIC 

BASIC  FOUR 
BUSIN'ESS  BASIC 

CJNICOMP 
COMP  16  or 
COMP  19  BASIC 

VARIAN 
620  or  V73 
BASIC 

G0SU3*- 

GOSUB  line  # 

GOSUB  line  f 

GOSUB  1. 

GOSUB  linc^ 

GOSUB  line' 

GOSUB  line^ 

GOSUB 

GOSU&OF 

GOSUB  fcjcp  OF 
line',  p^ram. 

GOflO** 

GO  TO  lin«  # 

GO  TO  line 

GO  TO  line  ' 

GO  TO  line  * 

GO  TO  line  * 

GO  TO  line  * 

GO  TO  line  f 

GOTO  OF 

GREATER** 

> 

SGT  or  > 

> 

> 

> 

> 

> 

GREATER  BQjllAL** 

>  ■  or  »  > 

\GE  or  >•  or 
»>  or>_ 

>» 

>  ■  or  ■  > 

>■ 

>■ 

HDLD 

IF** 

IF  exp  op  exp 
THES'  line  * 

IF  exp  op  exp 
THEN  line  ' 

IF  exp  op  exp 
THEN  line  * 

IF  exp  op  exp 
THEN  line  * 

IF  exp  op  exp 
statement 

IF  exp  op  exp 
THEN  line  * 

IF  exp  op  exp 
THEN  line  f 

'if  # 

see  READ  * 

or 
^*R1TE  f 

IF  MORE  ' 

_ 

40 


ERIC 


**CdnF!ands  «id  eloMnts  th»t  can  be  used. 


ISM  S3 
BASIC 

GB  2SS 
BASIC 

a^ic 

CCM-SIWRT: 
NTrt-BASlC 

hT^SriN'OOUSE 

&\s:c  11 

hisriNacusE 

B^\S1C  III 

GEI^RAL 

ADVAKCBD 
RASIC-16 

GOSUB** 

G06UB  lin«  ' 

03SUB  line  ' 
recursiocv 

GOSUB  line 

C0GU3  line  * 

005U3  line  i 

CDSyR  lin«  * 

OOSyH  line  ' 

GOSUBOF 

(Conputed) 

ON'  exp  GCSL^B 
line  ... 

ON  exp  GOSUB 
line  ... 

am** 

00  TO  lin«  « 

GO  TO  line  * 

GOTO  line  » 

GOTO  line  » 

GOTO  line  ' 

GOTO  line  1 

QTIO  line  i 

OTO  OF 

<coqpute<]) 

Oa  TO  line 
line  # 
ON  exp 

CN  e:q>  00  TO 
lint 

ON  exp  OOnO 
line  *p 

0«  exp  GOflC 
line  *i 

OS  exp  GOTO 
line 

OPJ  exp  OCHO 
line  ... 

ON  exp  GOTO 
line  ... 

GREATER** 

} 

J  - 

»  »  or  -  » 

*  »  or  *  » 

HOLD 

IF** 

IF  exp  op  exp 
THEN  iijie  ' 

IF  exp  op  «cp 
00  TO  line  ' 

IF  «>p  op  «xp 
T^ffiN  line  « 

IF  exp  op  exp 
THEN  line  * 

IF  exp  op  e>p 
OTTO  line  ' 

tr  CAp  op  CKp 
a)StlB  line  t 

IF  exx>  op  exp 
TIEN  line  9 

IF  exp  op  e>p 
OJID  line  ' 

GOSIB  line  t 

IF  exo  op  exp 
TKEK  statement 

IF  exp  op  «3ip 
TIEN  line  t 

IF  exp  op  exp 
THEN  linft  * 

IF  exp  op  exp 
THEN  line  * 

IF  eXp  op  exp 
QTIO  line  ' 

IF  EM)  t 

Ir  cSD  t  GXp 
line  mm 

see  ENDFILE 

IF  MORE  f 

**Cc<OTnds  and  eleiaonts  that  can  be  used. 


tiunr  AO  1 1  Art 
UNIVAL  IXw 

msic 

316,  S16,  ^id 
716  ar\S!C 

BASIC 

INIV  OF  MARYIAND 
RELEASE  V  1.5 

GCStlB** 

OOSUB  line  f 

00  SUB  line  f 

00  SUB  line  ¥ 

OOSUB  line  # 

G0SU3  line  * 

GObUB  line  t 

GOSUB  line  9 

OOSUB  OF 

(coqputei!) 

a)SUB  exp  OF 
lin*  *,  ... 

corro** 

GtnO  line  t 
or 

GOTO  line  4 

OOTO  line  ' 

corro  line  < 

GOnO  line  * 

OCrrO  line  < 

GOTO  line  t 
COtO  *  t  nuci 

GOTO  OF 

ON  exp  TtO  line 

4,  ... 

0>f  eitp  GOrio  line 
T  t  -  -  * 

o^'  txp  corro 

line  f,  ... 

ON  exp  OOTO 
line  ... 

ON  e>p  Gorro 
line  If  ... 

ON  en  corro 
line  ... 
ON*  ejcp  TIE* 

llllw  F  f    .  .  ^ 

GCFO  eitp  OF 
line  't 

ON  e)(p  THEN 
line  ... 
0.S  exp  GOTO 
li  no  ^  p  ... 

CREVTHR** 

exp  2)  or  > 

> 

> 

> 

.  »  or  GT 

> 

> 

GREATIft  EqUAL** 

CESlCejp  I, 
e^^i  2}or 
>  ■  or  »  > 

>  t*  or  ■  > 

>  ■  or  ■  > 

>  #  or  -  > 

GE  or 
>  ■  or  •  > 

>  #  or  *  > 

HOLD 

IF** 

IF  e^  op  exp 
TdEM  line  r 
IF  e^  op  tjcp 
COTO  line  ^ 
IF  exp  op  exp 
T10  statement 

IF  tjcp  op  ttp 
WES  line  f 
IF  ttp  c*.  exp 
GOTO  line  4 
IF  e^  op  e^ 
line  r  1, 
line  4  2, 
line  t  3 

IF  ttp  op  ttp 
TllEM  line  4 
IF  e^p  op  ttp 
GOTO  line  ' 
IF  exp  op  e^ 
HIEK  Statement 
IF       op  txp 
line  *  I, 
line  '  2t 
line  «  3 

IF  em  op  ttp 
liiK  f 

IF  ttp  op  exp 
THEN  line  4 
IF  eitp  op  txp 
GOnO  line  * 

IF  e>p  op  ttp 
imi  line  * 

IF  eip  op  exp 
IHEK  li.ne  4 
IF  eitp  op  e)(p 
corro  line  ' 

IF  END  i 

see  BNDFILE 

IF  EM)  '  exp 
IHEK  line  f 

IF  BID  t  nim, 
THEN'  line  f 

IF  END  '  num 
TTO  line  I 
IF  E:^  f  num 
GOTO  line  f 

IF  END  «  eitp 
THEN  lijw  4 

rFJTOE  # 

I?  rcriE  *  nura, 
HiEN  line  ^ 

TF  f^DilE  #  nwi, 
TriLV  line  ^ 
IF  ICRE  '  nwa, 
OJTO  line  # 

**CorTords  am<J  elements  that  can  be  used. 
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OOSUB* 


GOStB  OF 

Ccotputed) 


am)  OF 


CREATHR** 


CO 


GREATER  EQUAL" 


MICRCmTA 
BASIC 


GOIO  lint  t 


ON  exp  00  TO 
line  *, 


<l-HATA 
BASIC- 1 


OOSUB  lifie  f 


GOTO  line  * 


>  "  or  "  > 


OOSUB  line  r 


OOSlfB  cxp  OF 
line  *t 


GOTO  lim  r 


GCnX)  CKp  OF 
line 


KHVG  5300 


GOSUB  line  # 


GOfID  line  i 


GOflO  line  r, 
line  *  0!^  exp 


GENERAL 
MfXRX  I 


OOSUB  line  r 


00  TO  line  i 


Oh'  cjqj  GOno 
line 


4G 


GOSUB  liM  ^ 


oorio  line  I 


IDID 


IF* 


IF  e^  op  <0(p 
THEN  line  * 


IF  exp  op  exp 
THEN  line  1^ 


IF  e^  op  e>p 
TtEN  line  * 


IF  exp  op  exp 
TtIEK  line  ' 


IF  e^tp  op  exp 


IF  cxp  op  eJt^ 
THEN  lin*  # 


IF  END  f 


IF  EKD  f  exp 
THEK  line  ' 
ON  END  *  exp 
Tt^EN  lijve  * 


IF  END  I  num 
TtEN  line  ' 


IF  END  t  exp 
TtiEN  line  * 
IF  END;  exp 
line  r 


IF  M031E  I 


**Coflmnd5  and  elecctits  chat  can  be  used* 


BASIC  2.0 
SCOPE 

miV  OF  lOKA 

miA  GENERAL 

GE  mRK  II 
GEMARJC  111 

HPZOOOB 

HPZOOOC 

IF-THLN-EISE 

T^^^GE 

cot  usLsx; 

see  USING 

see  USIUG 

LMPLICATIC^' 

IMUT  varl,  ... 

iNrur  van ,  ... 

IKPUT  varl ,  ... 

IKtW  varl,  . . . 

INPUT  FS£M 

INPUT  LIKE 

see  iNFUr 

iSPVr  C-^  )  list 

LSTW  1 

IKPUT  #  QXp: 
varl,  varZp  . . . 

LVKJT  FILE 
Ee.i-pl  list 

IfJPUT  '  list 
iNTi;r  *  c^:  list 

"  '\  var,  ... 

INltGHR  ;ume 

Kja 

see  SCRATCH 

not  collect 

not  collect 

LAHGESr  1  ** 

£337 

7.2  B75 

1.70141  E38 

7.Z  E75 

1.70141  E3« 

£33 

E3a 

< 

< 

< 

< 

<  or  .LT- 

< 

< 

LESS  equAi^* 

<  p  or  ■  < 

<  *  or  *  < 

<  ■ 

<  *  or  .LE, 

<  m 

LET  ** 

LET  varl  »  varZ 
or  no  LET  " 

LET  varl,  ... 
var  n  ^  op  or 
no  LET 

LET  varl  ■  var2 
■  ^ , ,  exp 

LEI'  var  *  exp 
or  no  LET 

LET  vjrl  -  varZ 
■  var3  *  < .  "  txp 

LET  varl  ■  vw:2 
»  ...  -  trp 
or  jio  LET 

LET  varl  »  var2 
■  txp 
or  no  LHT 

AO 


I£M 
ITF 

PDP  10 

PDF  11 

UKrVAC  1100 
UEASIC  VERSiaV  2*0 

(^JLTICCMP 
Oft 

UNrV  KUSS 
E/ISICX 

xeox 

IP  stateiDcnt  op 
statencftt 
HEM  statement 
EL5E  statercnt 

sec  rnirtt  vOLtpj 

Sn  rKi«M  v^lNu 

sec  rKlM  USING 

see  FIELD 

see  PRINT  USING 

IMPLICATIOK 

IMF 

IMPCexpI,  ej(p2) 

INTUT  varl. 

INPUT  varl. 

IMVr  varl, 

IMUT  van, 

IKEVr  van.  ... 

LVPUr  varl  ^  <  <  < 

IKPUT  varl,  ,  „ 

INPUT  FRaJ 

LVPtrr  FRJOM  num; 
list 

INPUT  LLSTi 

INHn  LINE 
string  variable 

imxT  C  )  list 

see  I  SWT  FK>1 

INHJI  (61,  n) 
varl.  <<< 

Lspur  * 

GET  najne,  list 

IWVT  *K.  list 
INIur;  H,  list 

IKFtJT  *  e3<p, 
van,  varZ 

see  IKPUr  FROM 

IKIVT;  str; 
key,  list 

INPUT  "str^'i 
var 

IKTBGER  WHE 

nweric  i^auw 
followed  by  % 

KILL 

KILL  naiv 

KtlX  string 

see  SGIATCK 

LARGEST  *** 

E3S 

1.7  E3S 

1  ^  E38 

E3S 

7.237  B75 

LESS** 

< 

< 

< 

< 

LSSfcstplj  e3<p23 

< 

< 

LESS  BQU*\L** 

<  ■ 

<  ■ 

LEQCexpl,  e3<p2) 

<  ■  or  **  < 

LET** 

LET  van, 

LET  varl  ■ 
var  n  ■  eaqj 

LET  van  ■  . . . 
■  e^(p  or 
no  LST 

LE"'  varl,  varZ, 
*  *  *  var  n  ■  exp 
or  no  LET 

LET  varl  ■  var2 

LET  varl,  virZ. 
^ . .  var  n  -  exp 

or  no  LET 

LET  yarlf        ■  tvp 
or  no  LET 

SB 


Cosiun^s        «l*s«nt«  ttut  ean  used. 


IBM 
C,\LL/360-OS 

PDP  8/E 

ttOKErWELL 
200 

CDC  6000 
KIWOS 
ErXSiC  2,0 

NCR 
CENTURy 
200 

tiCSD* 
BASIC 
B6700 

KPZOOOF 

iF-TTiEN'-ELSE 

Stt  PRIMT 
USIXG 

FMT 

IMAGE 

IMPLICATION 

1  4^  ITU   1  V^liif^Wm 

fvpuT  varl 

INPUT  LIKE 

Lwn-  ( )  list 

see  READ  ' 

sec  RHAD  ^ 

see  IKTUT  i 

see  READ  t 

IKEirr  # 

sec  READ  ' 

see  READ  ' 

I\PUT  FILE 
(name)  varl ^ . , , 

see  RHAD  ' 

see  READ  ' 

see  READ  ' 

see  INPUT  t 

see  READ  ' 

lWrECER*WiE 

KILL 

KILL -name 
KIL-name 

LARuEST  '** 

IE  75 

IE  615 

IE  616 

IE  337 

7*2  E  75 

IE  47 

IE  3S 

LESS** 

< 

< 

< 

< 

< 

< 

< 

Li  Co  J 

K 

<  M 

<  V  or  ^  ^ 

<  M  or  < 

LET" 

LET  varl,* ,  * 
var  n  »  exp 
or  no  LET 

LET  var  *  exp 

LET  V3rl-var2=' 
,**v;iT  n-  exp 
or  no  LET 

LET  vaTl-var2* 
. ,*var  n-exp 
or  no  LET 

LBT  varl*exp 

LET  varl- , * , 
*vaT  n-exp 

LET  varl-  

var  n-exp 

sc 


*VKIVEKSITT  OF  CALUORH^A,  SAN  DIECO,  BURROUCHS  S6700 

ERIC 


NCR 
CENTURY  100 
BASIC  1 

BURROUGItS 
BSSOO 
BASIC 

BURROUGHS 
B2S00 
BASIC 

BURROUi^llS 
B5SO0 
BASIC 

BASIC  FOUR 
BUSINESS  BASIC 

UNICOWP 
COMP  16  or 
COMP  18  BASIC 

VARIAN 
620  or  V73 
BASIC 

 1 

  1 

I>«GE 

TABLB 

IMPLICATION 

1 



INTVT** 

INPUT  varl,... 

IKPUT  van  , .  . . 

IN'PUT  vai-1, .  .  . 

INPUT  varl . . 

IN'PUT  expl, .  . . 

INPUT  varl, . . . 

INPUT  varl,. 

Input 



IKPUr  LIN'S 

INTVT  C  )  list 

see  INPUT  ' 

sec  IN'PUT  i 

INJVT  ' 

l.^Pl^'  F1L& 
name  var  list 
INPUTfcxp, 
va  r  list 

•exp  J  cKR*nur.j 
FA"D=nucs)  expl , 

"""""""" 

ItSVT        var,  ... 

see  L^PUT  i 

see  1>;PUT  f 

isth:e(  N.\ME 

KILL 

—  *  -  - 

ERASE  name  * 

"  ~ "  ~  ~  ~ " " 

LARGEST  1 

IE  99 

4.314E  68 

1E99 

1E99 

1E99 

1.67E  73 

1E99 

LESS** 

< 

\LT  or  < 

< 

< 

< 

< 

< 

LESS  EQUAL** 

<  ■  or  -  < 

\LE  or  <  *  or 
"  <  or  i 

<  a 

<  « 

<  -  or  -  < 

<  ■ 

<  m 

LET  ** 

LET  var-exp 

LET  varl^varZ" 
. .  .var  n"  exp 
or  no  LET 

LET  var-exp 
or  no  LET 

Let  var-exp 
or  no  LET 

LET  var*exp 

LET  var»exp 

LET  varl*varZ* 
. . .var  n  ■  exp 

**Ca3ro3ivis  aiid  elements  that  can  be  vsed.. 


IBM  S3 

G£  ZSS 
TIMH  SIMRIW 

BASIC 

OM-SHARE 
SThBA^IC 

^'ESTLKaCJSE 
BASIC  n 

^^-ESriNQIOUSE 

BASIC  in 

GENERAL 
BASlC-16 

IF  cxp  op  exp 
HE^  stntcnent 
ELSE  st;itement 

IMAGE 

str  or  ksgc 

see  PRIfJT 

iriPUT  var  ... 

IKWf  var  ... 

ihWr  var  L,  .  ♦ . 

INTUr  var  ... 
or  ACCEPT 

IhlPlir  uar 

INPtJr  var  If  , ^ , 
rtr  iNPin*  t  uiTiut 

Ul     £tlf^v<           Ail  Lit*  w 

device  S  var  1 r 

l\TUT  FRCM 
cxp:    var  I»  ..v 

Tvpirr  (  1  list 

see  INPUT  FRCM 

INWT  File 
var  1 .  ... 

see  iNwr  FREM 

see  INJUT  * 

see  IKPUT  FROM 

nun  nam^ 

KILL 

LARGEST  **■ 

5E76 

5E76 

9.Z3E1S 

9.Z3E18 

9.Z3Eia 

LcSS*" 

< 

< 

< 

< 

< 

< 

< 

LESS  EQUAL"" 

<  ■ 

<  ■ 

<  ■ 

<  ■  or  *  < 

<  »  or  -  < 

LET** 

LET  var  1^  ... 
V3r  n  ■  exp 
or  no  LET 

LET  var  ■  exp 

LET  var  l»var  ;  ■ 
^ . .  var  n  »  ex[' 
or  no  LET 

LET  var  1  ^  . .  * 
=exp  or  LET  var  1 
var  2  ...  *  e^ 
Or  no  LET 

LET  var  1  ■  var  2 
»  ...  var  n  ■  exp 

or  no  LET 

LET  var  1  »  var  2 
-  . . .  var  n  -  exp 
or  no  LET 

LET  var  1  -  var  2 
■  . var  n  » 

S£ 


**Cocr^ds  and  eleoients  that  can  be  used. 


UNIVAC  1100 
UBASIC 

ta4E¥>4£LL  164(0 
XR/\SIC 

HCWE^TVELL 
316,  516,  and 

716  a\sic 

HCKEVKELL  ^00 
*  XBASIC 

H3NElf>i"ELL  600 
BASIC 

!{P20aOE 

UNTWC  1100 
USrV  OF  MARVLAKD 
RELEASE  \r  K3 

IF-IHEK-ELSE 

iF'exp  op  e>p 
THEN  statepent 
ELSE  Statement 

IMAGE 

see  PRIt^T  USIKG 

see  FIELD 

:  format 

:  foTTtat 

IMPLICATIOK 

J>tPCe)q>  Ij  2) 

var  It 

var  n 

Lvprrr  var  1 
var  n 

war  n 

^-ar  n 

[vpuT  vir  1 
^'ar  n 

IKPtJT  var  1 1  .  i 
var  n 

ispur  n^£w 

IN-Pin*  FR(M  cxp; 
var  If       var  n 

INPUT  LINE 

iNTUr  C  )  list 

see  INPUT  FHCM 

see  INPUT  f 

see  INPUT  i 

INPUT  # 

IKFtfT;  nam; 
var  1, 

INTUT  #  num. 
var  1  * 

INFVT  '  ntDiti 
list 

INPUT  "  var, 

^ee.iKfUT  FRCM 

seo  INPUT  * 

see  IKPUT  * 

s«  INPUT  * 

LNTECER  WtE 

KILL 

KILL  nantt 

LARGEST 

1ES8 

JE58 

JE38 

S.7S96E76 

t£38 

JE38 

tE3a 

LESS** 

LSS(exp  I,  e:q>  2) 
or  < 

< 

< 

< 

LT  or  < 

< 

< 

Lsss  eqjjAL** 

LBt}(e:q)  1,  exp  2) 
or  <  *  or  ■  < 

<  ■  or  ■  < 

<  ■  or  ■  < 

<  ■  or  ■  < 

LE  or 
<  ■  or  ■  < 

<  «  or  «  < 

LET** 

LET  var  1  ■  , ,  - 

var  i:  =  exp 
or  no  UET 

LET  var  ■  e)q> 
or  no  LET 

LET  var  J,  var  2, 
var  n  ■  exp 

or  ivo  LET 

LET  var  ■  e>p 
or  no  LET 

LET  var  1  «  x-^r  Z 
■  , , -  var  n  ■  exp 
Or  no  LET 

LET  v»r  1  ■  var  2 
■  „ ,  var  ^  ■ 

or  IVO  LET 

LET  var  1  ■  var  2 
■  , var  n  •  c>p 
or  no  LET 

**Ccnnanids  and  el<in&nts  that  can  be  ttsed. 


ERIC 


IF-THEN*ELSE 


IMACE 


IMPLICATIOK 


INPUT** 


INPUT  S:E£M 


HiRJr  LINE 


I^^pu^  ( )  ust 


iNEvr  t 


INPUT  "  *\  var. 


IPflECER  N^^tE 


KILL 


LARGEST  r** 


LESS** 


LESS  EQUAL** 


LET— 


HICRCtUTA 
GASIC 


IKFUr  var  1, 


1E57 


LET  var  »  cxp 


""Lorra^jJis  ana  elements  niat  can  w;  used 


BAStC-1 


INPUT  var  1, 
.  i .  var  n 


1E99 


<  ■  or  ■  < 


LET  var  »  exp 


HPSOQQ 


IF  e>p  op  ejqp 
TIO  state? c:*t 
ELSE  statencnt 


'WANG  3300 


IMAGE  formts 


li^PUr  var  1, 
>  i  >  var  n 


letter 


1E77 


LET  var  1  -  vnr  2 
*  -  var  I*  exp 
or  iw>  LET 


I  fomrtts 


INPUT  var  1, 
.  i .  var  n 


IE63 


LET  v^r  I,  var  2, 
, . ^  var  n  ■  tvp 
or  iw>  LET 


ELECTRIC 
MARK  I 


formats 


INPUT  var  1, 
-  *  -  var  n 


S.7S96()E76 


LET  var  ■  txp 


HANG  2200 


%  formats 


Input  var  i, 

var  n 


lElQQ 


LET  var  1,  var  2, " 
*  i  -  vaf  n  »  exp 
or  no  LET 


BASIC  2ja  , 
CDC  6600 
SCOPE 

lEM 
CPS 
UNIV  OF  I0«A 

JATA  GENERAL 

GO  mRX  II 
GE  MARK  III 

moooB 

IIP2000C 

LLS£ 

0  to  99999 

1  to  999 

1  to  99999 

1  to  9999 

1  to  99999 

1  to  9999 

1  to  9999 

LL\TUT 

LINIVT  *  K; 
list  of  str  v*ir 

LONGEST  STRING** 

72 

IS 

4095 

119 

72 

72 

MA5tGLS 

MARGIN'  4  N;  exp 

y^ARGIN  ^  expp  exp 
MARGIN  #  N, 
^tARGIN  ^  exp; 

f 

MAT 

MAT 

MAT 

MAT 

MAT 

MAT 

MAT 

MAXIMJK 

MAX 



MAX  ^UlRAY  SIZE 

500  eletacncs 

1024  eleincnts 

2500  elenents 

2500  elecKnts 

MAX  NESTING  LOOP 

10 

4 

20 

9 

9 

MAX  t  OF  DIM  IN  AN 
ARRAY 

2 

2 

2 

2 

2 

2 



MIN 

MIN 



MULTIPLE  STATB^TS 





NAME  AS 

NEXT  var 

NEXT  var 

KEXr  var 

KE>T  var 

KEXT  var 

NEXT  var 

NEXT  var 

Naa\TA  nua 
iXmtA  rILC 

IQ4 
in; 

LEASCO 

JD?  10 

PDP  11 

UN1V*\C  1100 
IfSASIC  \'nElSlfW  2,0 
>WCK;\TD  STATE  CiJG 

WJLTIOOM?  ^ 
OR 

UNP;  M\SS 
BASia 

XEROX 

Ll^JE  *** 

1  to  99999 

1  to  9999 

1  to  999$9 

I  to  32767 

1  to  99999 

1  to  99999 

1  to  99999 

LINPUT 

IXJWGESr  SIBIKG  ** 

18 

193  char 

size  of  core 

sire  o£  core 

S12 

30 

22 

KM** 

MAT 

MAT  ' 

MAT 

KAT 

MAr 

MAX 

MAX  ABSAX  size 

MAX  NESTING  tOOP 

IS 

Depends  on 
storage 

26 

^M  f  OF  DIM  IN  AM 

2 

2 

2 

Z 

2 

2 

2 

HXNDtH 

MIH 





HjtTlPLE  STATDEftS 

separated  by  ; 

KMME  AS 

KAME  str 

AS  str  protectior 

NEXT** 

NEXT  var 

NEXT  var 

NECT  var 

KEXT  var 

NEXT  var 

KEXT 

HECT  var 

NOQATA 

**CO«Mod«  «Qd  el*»eiit«  th«t  can  b«  u«ed. 

ERIC 


IBM 

CALL/360-OS 

PDP  8/E 

IIOS'EYWELL 
2O0 

CDC  6000  ' 
BASIC  2,0 

NCR 
CEtrruRY 

200 

ucsif 

BASIC 
B6700 

>IP2000r 

LLME  I** 

1  to  99999 

1  to  2046 

I  to  99999 

')  to  99D99 

1  to  9999 

ir  to  yyyy 

LINPUr 

18  char 

65  char 

72  char 

14  char 

IS  char 

72  Char 

HVT  ** 

MAT 

MAT 

MAT 

MAT 

MAXIMA 

MAX(varI,..0 

MAX 

£S , DOS  tes 

core  restricted 

core  restricte<3 

core  r^'Stricted 

4096  by  4096 

409S  by  409S 

4900  elements 

MAX  NESTING  103P 

IS 

a 

10 

10 

10 

no  limit 

9 

Amy 

2 

2 

3 

3 

2 

*  2 

2 

MiNCvari , ,) 



MIS' 



I.JflT'ft'TnT  E    i^fJl'l  LlidT^ 

HULTiPLc  oIATl^{Bfr> 

separ^tod  hy  \ 

NEXT** 

HPYV  var 

A  ca  I  V  3  r 

ifCAJ  Var 

I>CAI  vitr 

NTXT  var 

hcXJ  var 

NOD  AT  A  nario 
NODATA  FILE 

(narto) 

see  IF  ES'D 

*Vi'lViRSlTY  OF  CALIFORXTA,  SAX  DIEGO,  BUKROL'CHS  B67CC 
**Co=3:}c!$  tnc!  eiec«i)Ct  ChaC  cin  be  U3^< 


50. 


NCR 
BASIC  1 

BURROUGHS 

P  J  >  V  V 

BA*;  IC 

BURROUCKS 
BASIC 

BURROUGHS 
BASIC 

BASIC  FOUR 
BUSINESS  BASIC 

UNICOMP 
roMP  16  or 
COMP  13  BASIC 

VAR I AN 

620  or  V73 
BASIC 

LIKE  #  ** 

999999 

99999999 

9999 

9999 

1  to  9999 

1  to  9099 

1  to  0999999 

LiNPirr 

LONGEST  SnUNC  ** 

IS  characters 

IS  characters 
(in  PRINT) 

IS  characters 
(in  PRINT) 

:orc  deternined 

HAT  ** 

MAT 

MAT 

MAT 

MAT 

MAX  ARRAY  SIZE 

ISIZ  elements 

102Z  by  1024 

1000  elements 

1000  elements 

999  elements 

2SS  by  2SS 

MAX  NESTlfaC  UOT 

10 

core  dependent 

5 

5 

? 

MAX  t  OP  DIM  IN 

ARRAY 

2 

2 

2 

2 

3 

2 

3 

HINJHjH 

MJLTIPLE  STAlBieaS 

KVME  AS 

NEXT  var 

var 

AcAi  var 

J  var 

nOQATA 

6D 


**Cdnitan45  and  elements  that  can  be  used^ 


ERIC 


IK  S3 

MOD  6 
BASIC 

GE  2SS 
TIME  SliMtlJJG 

Ef\SlC 

I^ESTINGHOUSE 
BASIC  II 

)sE^INGHXJS£ 
BASIC  III 

t  GENIiRAL 

Aina*ATiav 

Br\SlC- 16 

0  to  9999 

I  to  99999 

0  to  99999 

1  to 

1  to  9959 

1  to  9999 

1  to  9909 

tiNwr 

18  Chirac tors 

IS  characters 

core  dependent 

core  dependent 

73  characters 

72  characters 

MARG1^' 



______ _  _ 

MAlT** 

H\T 

MAT 

Mat 

MAT 

MAT 



is  f\;3iction 

MAX  AI5RAY  3I2E 

set  by  system 
<lef ini  tion 

2074 

core  depended: 

core  dependent 

52,767 

^,767 

core  {dependent 

>1«  NESTI.VG  LOOP 

9 

26 

con?  dependent 

core  dependent 

no  liJtdt 

no  limit 

core  dependent 

MAX  *  OF  DIM  IN  AN 
ARRAY 

Z 

Z 

linited  by 
s^tensnt  length 

liudted  by 
line  lengCh 

limted  hy 
line  lertgth 

limited  hy 
line  length 

MLV  (var  1, 
is  £^ticn 

..^  

MULTIPLE  STATB-WT^ 

YES  )&  or,  or; 

rCAME  AS 

SECT** 

l^T  var 

S*BT  var  or 
SEXT  line  * 
or  NKCT  var,  » » » 

NEXT  var 

?CEXT  var 

**Corr::mds        elements  that  can  be  used* 


52 


UKIVAC  1100 

liD  ACf  ^ 

tCKDWELL  1640 

HCNEYKELL 
316,  516,  «Tbd 
716  EASIC 

HqKcVhELL  400 

HOKSfKELL  600 

HP2000E 

UKIVAC  1100 
UNIV  OF  MAR\XAKD 
RELEASE  V  1.3 

LINE 

0  to  99999 

i  to  32767 

1  to  9999 

1  to  99999 

1  to 

1  to  ssss 

0  to  99999 



LONGEST  SnUNG** 

511  characters 

ilepencts  on  core 

132  characters 

132  characters 

72  characters 

60  characters 

MARCm  t  met. 

MARGIN  *  nura, 
exp 

MAT 

MAT 

MAT 

MAT 

MAT 

MICCOW 

MAX(exp  1, 

MAX 

MAX  ARRAY  SIZE 

dete  mined  hy 
instftllftticti 

depends  oin 
available  core 

depends  on  core 

2000  elertents 
in  cor« 
220C:  on  disc 

depends  on  cone 
avxi  liable 

2000  elemnts      |  depends  on  core 

MAX  f^IHG  LOOP 

32 

S 

depends  on 
installation 

6 

26 

6 

32 

MAX  f  OF  DIM  AN 

Amv 

4 

2 

depends  on 
line  length 

depends  on 

2 

2 

2 

KIN  (txp  1, 
exp  2)  function 

Mm 

MULTIPtE  SlAtBerS 

separated  byt 

separated  by  \ 

tWE  AS 

NBCT 

NBCT  var 

KEXT  var 

KE)Cr  var 

NBCT  var 

NEXT  var 

KECr  var 

NEXT  var 

6F 


^■CdvBinds  and  elemnts  that  can  be  used. 


MICBOCMTA 
BASIC 

Q-tkMA 

HP3000 

>ttNC  3500 

GENEFtAL 
ELECTRIC 
>tARK  I 

KAjNJ  zzoo 

* 

LINE  *■* 

1  to  99^9 

1  to 

1  to  9599 

1  to  9999 

1  to  99999 

1  to  9999 

LINPUT 

LCNPtrr  string 

IS  characters 

IS  characters 

64  characters 

MARGIN 

MAT"" 

>iAT 

MAT 

mT 

^tAX  ASHAY  SHE 

depends  on 
core 

1S12  elements 

depends  on  core 

dimensions  ^255 

2074  elements 

diitier.sion£  ^  2S5 

MAX  t.tESTLNG  LOOP 

depends  on  core: 

10 

depends  on  core 

depOTwJs  on  core 

26 

depeivls  on  core 

K\X  #  OF  DIM  IN  AK 
ARRAY 

z 

z 

z 

z 

2 

MTN 

fULTlPLE  STATEMENTS 

in  LET  stateii«nt 
separated  ly  , 

separated  by  t 

sep*tated  by  ^ 

»1AME  AS 

4  Va  * 

WEDCT  var 

mjhata 

"*Ccirrands  and  eleasen^s  that  can  be  used. 


BASIC  2.0 
CDC  6600 
SCOPE 

1^ 

CPS 

1 

1 

lATA  GENERAL 

GE  M\RK  II 
GE  ^!ARK  III 

HP2000B 

HP2O00C 

NOT 

_  _  _  _  ^ .  _  _ 

WOT 

NOT 

WOT  EQUAL 

<>  or  >< 

^  -  or  <> 

<> 

<> 

<>  or  ..SE. 

<>  or  # 

<>  or  i 

.M^tHRlC  VARIAiJLE  UAMB  ** 

letter  or  letter 
digit 

letter  or  letter 
alpha,*:)cric 

letter  or  letter 
digit 

letter  or  letter 
dig.t 

letter  or  letter 
digit 

letter  or  lettei 
digit 

letter  or  letter- 
digit 

OH 

nuRij  . . . 

ON  exD  GOTO 

line  *p  line  f 

OX  eip'cOSUB 
tmu  'f  line  ^ 

Gyp  GOTO 
line  ... 
ON  exp  TiEN 
lini*  ... 
ON  <?xp  GOaJS 
line  ... 

ON  exD  GOTO 
lir^  ... 

CN  HtBOft  GOTO 

OPEN 

sec  READ  or 
WRIT*  PiLH 

soe  PILH 

OPITJ  PILE 
nUkHf  ntn^f  fiaine 

OPEN- nans,  nm 
OPE  -nane,  nira 

OPEfJ-uac*,  mm 
OPE  'Jiasft,  nisi,  msi 

OR 

OR 

PAU5E 

?tm 

PRIKT  list  of  e^ 

PRIST  list 

PRINT  list  of 
e:(p 

PRI.ST  list  of  e^q) 
or;  list  of  exp 

PSU.ST  list 

PRINT  list  Of 
exp 

PRINT  list  of  ejq? 

PRINT  i 

see  KKITE  FILE 

PRIST  *  e;qj: 
list  of  exp 

PRL\T  FIl^  (expl 
lis:,  of  exp 

PRI.ST  F  exp,  list 
PRINT  1  exp:  list 

PRIST  1  exp: 

list,  BtD 

DO  Tift  1  evaY- 

list,  END 

PRINT  '  expl; 
PRl.ST  1  exp:  list. 

PRl.vr  *  USING 

P?I.vr  ^  exp: 
USING  str  exp. 
Us: 

PR1\T  FT'.£  Texp) 
irs'NG  "s^riiip", 
liii 

PRIST  ^  exp, 
U5KG  str,  lift 

**Co^inH3  ind  elecMRCs  thit  e«n  be  used. 


194 

iTr 

I^ASCO 

PDP  10 

PUP  11 

usrwx  1100 

lUBASIC  VnRjRlON  2.0 
MANKAtO  state  tUi 

.^flJLTlCCMP 
OR 

UW  Ma^<v 

a\sia 

NOT 

NOT 

NOTCexp) 

NCFT  BQtlU** 

<> 

<> 

<> 

<> 

NiaqCejcpl,  exp2> 

<> 

<>  or  >< 

^WER1C  VARIABI-H 

letter  or  letter 
digit 

letter  or  letter 

letter  or  letter 
digit 

letter  or  letter 
digit 

letter  or  letter 
digit 

letter  or  letter 
digit 

letter  or  letter 
digit 

CM 

GOTO  varl,  ... 
var  n  ON  eJtp 

ON  exp  GO  TO 
line  ... 
ON  ejq)  THEN 
line  if  ... 

ON  e3<p  GOTO 
line  ^ f  line  ' 

as  exp  ooajB 
line      line  * 

num  GOTO  line  ' 

ON  exp  GOTO 
liru^  Kf  line  ' 

0«  esqp  GOTO 
1 ine  * ,  ... 

OH  mo^  oaro 

ON  I31R0R  GO  TO 







OPEN 

0P3*-nane,  mm, 
nun 

OPEN  najne 

FOR  ''"^M 
DUTTUT 

OPDJ  name 
SWIBOLIC  iNiur 
BIKAfty  OUTPUT,  nuci 

0PE4      name,  n 

0?Bt  name,  TO: 
Str,  GET,  T  FILE 

OR 

OR 

OR 

IORCe>pl ,  e>^2) 

PAUSE 

PAUSE 

PAUSE 

PLOT 

PLOT  enpl  ■ 
'^exp2'* 

PRINT** 

PRINT  list  of  e?qi 

PRIOT  list 

PRINT  list 

PRINT  list 

PRINT  list  of  typ 

PRINT  list  of  e;q^ 

miNT  list 

PRINT  » 

see  WITE  ' 

PRINT  i  cxpt  ejq>; 
list  BiD 

PRINT  i  N,  list 
PRINT:  N\  list 

PRINT  '  exp,  list 

PRINT  OH  nun:  list 

sec  t^TUTE 

PRINT:  twi\  Vey, 

115 1 

PRINT  1  USING 

PRINT  USIN-S 

string 
i  vATf  line 
list 

PRINT  *  e>p, 
USiNf:  sir  exp, 
list 

see  KRITE 

7B 


'*Coc3rind«  «n<l  el«iiicnc«  th«C  c«n  be  u«e<!. 


ERIC 


CALL/36D-0S 


PDP  3/B 


HONEYWELL 
200 


CDC  6000 
EAS7C  2.0 


NCR 
CENTURY 
200 


UCSD' 
BASIC 
&67D0 


7C 


HP2000F 


NOT 


NOT  EQUAL* 


<>  or  / 


<>  or  X 


<>  or  >< 


c>  or  ^ 


m-vmC  VARIABLE  NAME* 


letter ,S ,^ ,'or 
digit 


letter  or 
letter  digit 


letter  or 
letter  digit 


letter  or 
letter  digit 


letter  or 
lette^r  digit 


letter  or 
letter  digit 


letter  or 
letter  digit 


ON" 


ON  exp 
GOTO  litie' , . 
linetf 


ON  exp 
Go  TO  line^ 
linc# 


ON  c>cp 
GOTO  litie',. 

OK  exp 
Ti;SN"  line^, . 


CN  ERROR  GOTO 


OPEV 


OPEN  exp,var, 

INPUT 
OPEN  exp.var, 
OUTPUT 


OR 


PAUSE 


Pause  comment 


OPEN-name  ,nuot, 
OPE-name,nun, 


OR 


PLOT 


PRINT  ** 


PRLVT  * 


PRINT  list 


see  KRITE  ' 


PRING  list 


PRINT  list 


see  Write  ' 


PRINT  list 


PRINT  FILE 
(nanc)  list 


PRINT  list 


PRIK'T  list 


PRINT  *  exf 
list 


PRINT  list 


PRIST  *  expi 
list 


PRINT  f  iJSlNC 


ERIC 


Q  *UK1VERS1TY  OF  CAUFORNlA,  SA«  DlEOO,  EUm^JGKS  1670^ 

**C<?^^ind3  and  «I«iE«Dts  thAt  cm  b«  n^^d. 


KCR 
CE^*TURY  100 
BASIC  I 

BURROUGHS 
BSSOO 
BASIC 

BURROUGHS 
B2S00 
BASIC 

BURROUGHS 
33SO0 
BASIC 

BASIC  FOUR 
BUSINESS  ElASIC 

UNICO^fp 

COMP  16  or 
COMP  IS  BASIC 

VARIAN 
620  or  V73 

BASIC 

!COT 

*10T  BqUAL" 

<> 

\NF  or  or 
X  or  y 

<> 

<> 

<>  or  X 

<> 

<> 

hKNERlC  VAREABLE  NAME  ** 

liitter  or 
lettc'r  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter 
letter  digit 

letier  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

ON 

O^f  exp  GOTO 
line  . 

OX  var  GOTO 
1  inc  '  , » . . 

ON  ERRDR  OOID 

ON  STATUS 



........ 

OPEN 

OPEX  tnus;) 
itar^e 

OR 

OR 

PAUSE 

VtA It  e xp 

PLOT 

PRINT 

PRIKT  list 

PRINT  list 

PRIKT  list 

PRINT  list 

PRIKT  list 

PRINT  list 

PRINT  list 

PRINT  * 

PRINT'  exp,list 
PRINT  FILE 
itaDe  t  list 

I>RI^T  (ni=n,IMKexf 
list 

PRINT  1  USING 

see  I^RITE  ' 
USING 

7D 


••Ccrrands  ajtd  elenents  tliat  can  be  used. 


im  S3 
HDD  6 
BWIC 

C£  2SS 
TIME  SirVftlWC 
BASIC 

COM- SHARE 

NTWiWSiC 

WESTINGBDtjSH 
HA?5rC  11 

KESTINGtt:USE 
BASIC  HI 

AlfPTMATK^' 
Air/ANCED 
SIC- 16 

NOT 

NOT 

WOT 

SOT 

NOrr  BQUAL** 

<> 

<> 

<>  or  t 

<>  or  ><  or  t 

<>  or  ><  or  ' 

<> 

NIMERIC  VARIABLE  NAME** 

letter,  @,  f,  $; 
or  3  letter^ 
*,  or  S  difiit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

ON  exp  GOTO 
line  *j 

see  GOTO  OF 

0^'  exp  COTO 
line  *y 
ClY  e?^  GOSUB 
line  , 

ON  exp  GOTO 
line  ''j 

ON  exp  GCVro 

:  lino 

ON  exp  COfTO 
line  *^ 

ON  ERROR  OOro 

ON  SIROR 
GOTO  line  * 

OPEN 

ofTPyr 

OPKJ  f  name! . 
INTVr 

OPE.V  Ifilei. 



optionSf  exp 



OR 

OR 

OR 

OR 

OR 

PAUSE 

PAUSE  cocmcttt  or 
PAUSE  or  SUSP0CD 

PAUSE 





PLOfT 

PRINT  list 

PRINT  li#t 

PRINT  list 

PRINT  list  or 
DISPJAY  or 
OUTP'*"  or  TiTE 

PRINT  list 

PRINT  list 

PRI^^^  ust  or 

PR1^*T  $  output 
device  $  list 

PRINT  t 

srje  IsTtlTE 

PRIN'T  FILE, 
var  1» 

PRINT  CN  exT>j 
list 

PRim*  t  USING 

PRINT  IN"  FC*>? 
str  0^:  exp  1»  | 

1 

1 

7E 


'Coan;at)ds        eleinents  that  ca.<i  be  used. 


UNIVAC  1100 

HDHEiwELL  1640 
XB^IC 

lOJEYWELL 
316,  Sl6,  nnd 
716  BASIC 

IfKElfKEII,  600 

f!P2000£ 

UNIVAC  1100 
UNIV  OF  mRYlAND 
RELEASE  V  1.3 

NOT 

NOT 

NOrr  EQUAL** 

NGQCexp  1,  exp  2) 
or  <>  or  >< 

<>  or  >< 

<>  or  >< 

<>  or  >< 

KE  or  <>  or  >< 

1  or  <> 

<>  or  1 

Nl^fERlC  VARIABLE  NAME** 

letter  of 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  ^igit 

letter  or 
letter  digit 

let:er  or 
letttrr  <^igit 

letter  or 
letier  digit 

letter  or 
letter  <Jigit 

OH 

a^!  exp  GOTO 
line  ' r  -  -  - 
exp  THEM 
line  J' ,  ... 

ON  exp  GOTO 
li ne  if  .  1 . 

ON  exp  GOTO 
1  ii*e  *  > 

0.*^  exp  GOTO 

1  iFio  '  >  . .  * 

OM  exp  HSM 
liiw  ' , 
ON  ejcp  GCTO 
line    J  ... 

o»  exp  innN 
i*> t  ^ )  ■ ■ ■ 

0«  exp  GOTO 
line  ^,  ... 

CN  ERHOTt  GOTO 

OPE* 

OPEN  none  FOft 
options  A  FILE 

see  FILES 

see  FILES 

see  FILES 

OPEN  nam,  num 

OR 

lORCexp  1>  exp  2) 



OR 

PAUSE 

PAUSE  or  BRK 

—  

PLorr 

PRIM^* 

PRI.W  list 

PRINT  list 

PRIWT  list 

PRINT  list 

PRHT  list 

PRIST  list 

PRINT  list 

PRI^T  1 

PRIST  CN  exp: 
list 

see  MtlTE  ^ 

PRINT:  naiw: 
list 

PRINT  ^  mn, 
list 

PRIST  '  mjij 
l:5t 

PRI.ST  ^  exp; 
list 

PRI^T  *  USING 

see  PRINT  USING 

see  MinB  I  USI.^G 

PRIST  '  mci, 
USING  list 

PRIST  f  tivrif 
USING  mm, 
list 

**Coasands  and  elements  that  can  b«  used^ 


7F 


ERIC 


MlCSDOflTA 
BASIC 

BASIC-1 

HP3000 

3300 

CE>ERAL 
ELBCmiC 
I 

KANX;  2200 

Hcrr 

}KfT  EQUAL** 

f 

<>  or  >< 

f  or  <> 

<> 

o 

<> 

NUMERIC  VARIABLE  WM^*  ^ 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

letter  or 
letter  digit 

ON 

ON  exp  0310 
line  *p 

s«  CCfTO  OF 
see  GOSIJ&  OF 

see  GOTO  OF 

ON  e^p  OQflO 
lljie 

ON  QIRDR  OCnO 

OPEN 

see  ASSIGN 

see  FILES 

see  FIIXS 

see  ASSIGM 

Oft 

OR 

OR  Cexp 
e;tp  ji) 

PAUSE 

PLOT 

mta** 

PRINT  list 

PRINT  list 

PRINT  list 

PRIKT  list 

FRIKT  list 

PRINT  list 

PRINT  1 

PRINT  '  ejcp; 
list 

PRINT  *  expp 
exp  1;  list 

PRIMT  #  USING 

'Cooinandls  and  elements  th^t  can  be  used.^ 


(DC  6600 
SCOPE 

im 

CPS 
UNIV  OF  lOHA 

GE  Mm  III 

PRINT  USIKG  Iji, 

PRINT  USING 

PRINT  USM 
**strin5i**t  list 

PRINT  USING 

PRINT  USISG 
line  list 

PRINT  USIHG 
String  expf  list 

RAKDCMIZE 

RANDOMIZE 

RANTOM 

RtaN 

READ** 

READ  varl, 

READ  vart,  ... 

READ  varl,  - . . 

READ  varl^ 

READ  varl,  ... 

READ  varl,  ... 

READ  voTlw  . . . 

exp,  list 

READ  f 

READ  FILE  (name) 
list 

RBAD  »  <i3cp; 

READ  FILE  lexp] 
V2rl ^  ... 
READ  FILE  texp, 
exp]  varl^  ... 

READ  *  mn^  list 

K1;AU*  Jiunt,  list 

READ  ^  e^; 
list 

READ  '  e>j>,  exp; 
list 

READ  i  exp;  list 
READ  f  aiop  r  cxp  t 
list 

READ  C  ,  ) 

see  READ  f 

sec  READ 

see  READ  f 

RELEASE 

R&l** 

R0I  messa^ 

R0I  message 

R0I  qtcssage 

REM  message 

RBf  messa^ 

R&f  messa^ 

RBi  i^sage 

RESET 

see  RESTORE 

RESET  *  ejqp: 
exp 

see  REmlSD 

see  REKIKD 

RESTORE** 

RESTORE 

RESTORE  FILE  (mums) 

RESTORE 

see  RESET 

RESTORE 

RESTORE  f  ejqp 
RESTORE 
RESTORE:  exp 

RESTORE 
RESTORE  line  ' 

RESTORE 
RESTORE  tine  ' 

RESUHE 

REIURN"* 

REIURW 

RETURN 

RETURN 

J 

RETURN 

RETURN 

REIURN 

**COraand*  ADd  «leo*nt3  that  c^n  te  used* 


IB4 

LEASOO 

FDP  10 

POP  11 

UNIVAC  1100 
UE,^S1C  VERSION  2.0 
MANKAtO  STATE  CLG 

OR 

UNIVfiASS 
EASICX 

imj*r, USING 

nSHT  USING 
nun,  list 

PRIWr  USIKG 
nun,  list 

PRINT  USING 
string  trp,  list 
PRINT  USISg 
num,  list 

PKlWr  USING 
string  ei^,  list 

PRlJfT  USIKG 
n,  list 

PRINTUSING 
nuQp  list 

RAJ5XW 
RrWDOMIZB 

RAM»tIZ£ 

RANDCMI2E 

RANDCWIZE 

READ** 

HEAD  list 

READ  v»n,  ... 

READ  varl,  ... 

READ  varl, 

READ,  varl,  ... 

READ  varl, 

READ  van,  ... 

READ  f 

READ  f 

list 

READ  f  X,  list 
READ:  N,  lift 

see  INUTT 

see  READ  (  ) 

GET;  im;  key, 
list 

READ  f  .  1 

see  INTUT 

READ  (60,  n) 
list 

RELEASE 

JCILL 

RELEASE  u 

RBt** 

iDes£«g« 

RBJ  message 

RBf  ncssjigc 

RB{  messa^  or  f 

REM  or  f  riesfagi^ 

RDl  message  or  «t 
wsss^fie 

RESET 

RESET  r\am 

see  RESTORE 

SM  RESITS, 
SETFTR 

RESTORp 

RESItRE 

RESTORE 
RESTORE  line  * 

RE^RE  list 
RE5TORB  • 

RESTORES 

RESTORE 

RESTORE 
RESTORE  • 

RESTORE 

RESTORE 
RESTORE  lijie  f 

RESU4E 

RESUt^  line  t 

REIUIW** 

RETURN 

RETURK 

RETIIW 

RHjURK 

RETURN 

ICIURK 

IBH 
CALL/360-OS 

PDl^  S/E 

HONEYWELL 
200 

CDC  6O00 
BASIC  2.0 

KCR 
CENTuRi 
ZOO 

UCSD" 

RyVSlC 

B6700 

HP2000F 

PftlKT  USISQ 

PRINT  OSlKG 
Line t  fOxpl t ' ' * 

PRINT, line', 
varl , ... 

PRINT  USING 
strinf ivarl . . 

RAM)0MIZB 

RANDOMIZE 

RANDOM 
RANDOMIZE 

READ  varl,. .  * 

READ  varl,. . . 

READ  varl.... 

READ  varl,. . . 

REA:j  varl, . . . 

READ  varl^--. 

READ  varl*. .- 

READ  FORWJtD 

R£AD  r 

GET  exp: 
v^irl  1 . . . 

READ  #  exp, 
var I , . . . 

READ  FILE 
(name)var 1 -  - 

READ  *  exp J 
varl* . . . 

READ  f  exp, 
varl » . . . 

READ  (  ,  ) 

see  RCAD  r 

see  READ  ' 

see  READ  ' 

see  READ  f 

see  READ  I 







REM  message 

REH  ciessage 

REM  message 

REM  ncssage 

REM  message 
REMARK  message 

^  REM  message 
REMARK  ncssagc 

REM  message 

RESET 

RESET 
explf . . . 

see  REKIND 

RESTORE  FILE 
(name) 

RESTORE  '  •xp 

READ  f  nun,  1 

RESTORE** 

RESTORE 
comment 

RESTORE 

RESTORE 

RESTORE 

RESTORE 
RESTORE  * 
RESTORE  $ 

RESTORE 

RESTORE 
RESTORE  no a 

RE5U^!£ 

REIURN** 

RETOR.V 
consent 

RETURN 

RETUR.S* 

RETURN 

RETURN* 

RETURN 

RETURN 

*fMVERSnT  OF  CALIFORSIA*  SAK  DTECO,  BLUROLXHS  B67O0 
*^Coc7^nds  ind  elencnCl  Chat  cin  be  used-^ 


KCR 
CENTURY  100 

BURROUGHS 
B5500 

BURROUGHS 
SASIC 

BURROUGHS 
B3500 
BASIC 

BASIC  fo>;r 

BUSINESS  BASIC 

UKICOMP 
Cun«    *  V  or 

VARI*-** 
Obk/  or  V/ J 
BASIC 

PRI?fT  USING 







READ** 

READ  varl^ . . . 

READ  varl ^ . . . 

READ  varl  ^  ... 

READ  varl,  . .  * 

READ  varl#* . . 

READ  varl  • . . . 

READ  PDRKAW) 

READ  ' 

see   INPUT  ■ 

READ  Crtum^IXD- 

w  A  p  J  C  T\t\    nu  In  J 

END^'non) 
list 
or 
EXTRACT 

READ  f  *  ) 

see  INPUT  ' 

see  re^id  ' 

........ 

RELEASE 

ERASE  name 



RBI  ** 

REM  message 

REM  message 

6  :rtessage 
REM  message 

d  mess^ige 
REM  inessage 

:  mess^ige 
REM  oessage 

REM  message 

REM  message 

RESET 

RESTORE 
FILE  nane 
RESTORE  '  ^JCp 

RESTORE  ** 

RESTORE 

RESTORE 

RESUME 

RESUME  line  ' 



RglURN  ** 

RETURN* 

RBTURN 

RETURN            j  RETURN' 

RETURN            j  RETURN 

RETURN 

**Ccn»nands  aixJ  elements  that  can  be  used* 


IBM  S3 
IttSIC 

C3;  zss 

TIME  SHARIKG 
BASIC 

HASIC 

CCM-SJ'ARE 
KD^EASrC 

hTSriNtlCUSE 
BASIC  II 

WESTIN-GHOUSE 
BASIC  III 

CBCERAL 
iWJTCKATrON 
AP/ANCED 

PRINT  USING 

PRINT  USING 
line  S  list 

SET  DIG  ITS 
SET  FORM,\T 
not  really 
fOTTOtted  prirt 

FRl.vT  I.S 
FORM  str  exp: 
exp  1,  ... 

RANDCMIZE 

"  ~  "  "  "  "  "  " 

READ** 

READ  var  1^  ... 

READ  var  1,  ... 

RI:aD  var  1  p  .  . 

READ  var  1 ,  ... 

READ  var  I,  ... 

R£\D  var  1,  ... 

READ  var  Ip  ... 



READ  f 

GET  naw,  var^ 
var  1,  ... 

READ  f  exp, 

var  1,  var  2,  ... 

RB^)  FILE 
var  1 ,  ... 

see  INTUT  FRO^i 



REM)  C  ,  ) 

see  READ  * 

see  READ  f 

see  READ  ^ 

see  l.STUT  FRCM 

ElELEASE 

see  I.STUT  FKM 





RIM  coiyment 

RE^^  coixicnt 

RIM  ccmsent  jr 
!  coimtcnt 

R&I  cccnenT  or 
\  co:iincr.t 

REM  cortwnt 

R31  co^mnt 

R&J  cornent 

RESET 

RcSET  naire,  . . 

see  RDflQ.ST) 

RESTORE** 

RESTORE  coTjaent 
or 

RESTORE 

RESTO.RS 

RESIORE 
RESIORE  * 
RESTORE  * 

RESTORE 

RESICRE 

RESTORE 

RE  TORE 

ftESlME 

RESME  or  CO 

PROCEED  AGAIM 

REIURV** 

REIUR.S'  cocrxjnt 

or 
RETt'RN" 

RETURN* 

REJVRN'^ 

RETUR.S' 

REIVRS 

RETURN 

RETURN 

8E 


**Coi;:':wkr)ds  and  cletrents  that  can  be  used. 


UNIVK  1100 
UESASIC 

HOPEWELL  1640 
XBASIC 

316^  S16^  and 

7i*i  a^sic 

HDNEWELL  400 
XhASlC 

IttiEifWELL  600 
BASIC 

HP2000E 

UNIVAC  1100 
UNIV  OF  MARVLV® 
K£I£ASUV1,3 

PftlNT  USING 

PRINT  ON  ejq> 
var  1^ 

PRiNTfr  nun,  list 

PRINT  USI^^G 
ntn^  list 

PRINT  USIKG 
no»j  list 

RANEXMIZC 

RA^NDOMIZE 

READ** 

READ  var  1, 
var  n 

READ  var  1  ^  Hit 
var  n 

BEAD  var  1^ 
var  n 

READ  var  1 ^  ... 
var  n 

READ  vrar  ... 
var  n 

READ  var  1^  ... 
var  n 

READ  var  1^  .. 
var  n 

READ  FOftiWlO 

READ  ¥  ejq>,  exp 

HEAD  i 

see  INPUT  FRCH 

READ  f  e^, 
var  1,  var  2, 

READ  P  nin^ 
vnr  It  ... 

READ  '  n\jct,  list 
READ:  mim,  list 

READ  *  exp; 
var  1,  ...  var  n 

READ  C  ,  ) 

see  INPUT  FKCM 

set^  READ  ^ 

s«c  READ  f 

SCO  READ  1 

sc^  READ  f 

RELEASE 



RBI** 

RBI  connent  my 
follov  statements 
after  special 
character 

RBI  comnent  or 
*  coCTiient 

RBI  corvient 

R&l  cociront 

RBI  coiment 

RSl  cotsBC^t 

Rat  connent  olj/ 
follow  statements 
after  special 
character 

RESET 

see  RD/INE) 

s«e  REWIND 

READ  ^  cxp^  1 

RESTORE** 

RESTOTE  or 
RESTORE  *  or 
RESTORE  S 

RESTORE 

RESTORE 

RESTORE 
RESTORE  * 
RESTCj^E  S 

RESTORE 
RESPOStE  * 
RESTORE  * 

RESTORE 
RESTORE  line  f 

RESTORE 
RESTORE  * 
RESTORE  i 

REStftE 

REIURN***  j 

R^^^T^f^ 

RETURN 

REnmS" 

RETUW 

RETURN 

I 

RETURN 

**Cocnaf:ds  and  elements  that       be  used. 


miNT  USING 


MICRODATA 
BASIC 


Q-a\TA 
SASIC-I 


mr»;  list 
PRIST  USING 
stv  varp  list 
PRINT  USING 
5tr:  liSt 


WAW  3300 


PRI^T^  usim; 

nt£n,  list 


ELDCTHIC 
MARK  I 


PRINT  USING 
rujip  list 


ypSG  2300 


PRIST  USING 
nurip  list 


RAMXMtZE 


RANTOMIZE 


READ** 


READ  var  1, 
var  n 


READ  var  1 , 
var  n 


READ  var  1^ 
var  n 


READ  var  1  ^ 
v?ir  n 


READ  var  1 , 
var  n 


READ  var  I, 
var  n 


READ  FORKARD 


ADVANCE  *  trxp; 
C'^cp,  var 


READ  r 


fVRU)  '  exp;  var  1 

READ  ^  cxpp  exp; 
\'ar  1  p       p  var  n 


FILE  READ  I  ntfft, 
var  1, 
var  n 


UEAD  P  exp, 
var  2,  . ,  ^  var  n 
READ;  expp  var  1^ 
^  ^  ^ ,  var  R 


DATAUW) 


READ  (  ,  ) 


RELEASE 


RBI** 


see  READ  it 


see  READ  ' 


see  JtBM)  ^ 


R&!  eoitmcrt 


RSf  ccnmcnt 


RBI  eoraicnt 


R&l  eocrert 


REM  coiiTJ»nt 


RESET 


see  RB^IKD 


RESTORE^* 


RESTORE 


RESTORE 


RESXRB 
RESTCRH  Jiun 


RESTORE 
RESTORE  num 


RESIDRH 


RESTORE 
j^ESlORE  nan 


RESUME 


RES^fE 


RSIURN"** 


RETURN 


RETl^TW 


REKmK 
RETUIW  QVp 


REIVRX 


RETURN 


**CoiiRand£  and  elements  thit  <^ai^  be  used. 


E\SIC  Z.O 
CDC  6600 
SCOPE 

ra4 

CPS 
UMY  OF  lOflA 

lATA  C&aiR.U 

\ 

CE  MARX  m 

r0>;OOOB 

HP200GC 

READ  '  t^Jtp,  1 

SCRATCH 

SCRATGI  *  <txp 

SCEATQI  *  t-^ip 
SCPATCII:  exp 

SETFTR 

SEr*^  ocp  TO 

see  READ  ^ 

ice  READ  ^ 

atUUEST  *** 

E-36e 

l\4  E-79 

1*46937  E-39 

)  .J^p;>7  E-39 

E-3S 

STOP 

STOP 

SHOP 

STOP 

STOP 

STRING  QOTES** 

1 1 

1 

1  1 

*  * 

STRINGS 

STRING  VAKlABiE  KAME  ** 

letter  J 

letter  J 

Nira  name 
follow^  by  S 

letter  J 

letter  J 

letter  J  _ 

letter  * 

SUB 

SUB  nare; 

TIME 

TIME  n 

UNLESS 



USER  pEFI^a3)  function 

FN  letter 

FN  letter 

FK  letter 

FN  l;tter 

FJ)  letter 

FN  letter 

FN  letter 

;2 


lEK 

ITF 


PDP  10 


PDP  11 


UNIVAC  1100 
SttKKATO  STATE  CLG 


WJLTIOCMP 

OR 

EASia 


XEBOX 


96 


SCRATCH  list 


St«  KILL 


SCRATCH  nuci 


SETPIK 


SET  N,  C)q> 


»ttLL£ST  t** 


^,4  E-79 


E-3S 


.4 


,14  K-33 


E-99 


J,3t?  E-79 


STOP* 


STOP  or  STOP 


STOP 


STOP 


STOP 


STOP 


STOP 


STRING  QUOTTES** 


STBLVGS  nun 
default  eO 


STOIKG  VARIABLE  fitolE 


SUB 


letter  J 


Metric  niim  $ 


letter  S 


SOBBO) 


TIME 


UNLESS 


UNTIL 


USER  DEFIMED  IWHTCK 
MMES  ** 


EK  letter 


t74  letter 


FN  letter 


m  followed  by 
variaM^ 


FW  letter 


FK  letter 


Bi  letter 


**Co=mjn<Ij  and  eleaencj  that  ean  be  uaitd. 


IBM 
CALL/360-OS 

PDP  S/E 

HONEYIVELL 
200 

CDC  6000 
BASIC  2.0 

NXR 
CEXTl'RY 
ZOO 

ircsp* 

BASIC 
B6700 

HP2000F 

REWLND 

See  RESET 

RESTORE  '  exp 

sec  RESET 

see  RESET 

see  RESET 

SCRAlCil 

SCRATCH  '  exp 

_______ 

SETFIR 

lE-73 

lE-615 

1 

lE-616 

1E-36S 

1E-3B 

STOP'* 

STOP  coinnent 

STOP 

STOP 

STOP 

STOP 

STOP 

STOP 

snuNG  (jxriBs** 

or  * 

tt 

11 

It 

It 

tt 

STRINGS 

$TRl\G  VARIABLE  ^AME** 

letter  S 

N'unieric  narae  * 

letter  S 

N^umeric  n^ne  > 

letter  S 

SUH 

- 

TIME 

UNLESS 







UNTJL 

USER  OEFI^JEO  FUKCTION 
\AMES  ** 

FN*  I<;tter 

letter 

letter 

FN  letter 

Fx  Itrtter 

FN  letter 

*U:;iVESSlTY  OF  CALlFORSlAt  SA*J  DlECO^  BURROUCHS  B670O 


74. 


NCR 
CENTURY  100 
BASIC  1 

BURROUGHS 
BSSOO 
BASIC 

burroughs 
b:soc 

BURROUGHS 
B3SO0 

BASIC  FOUR. 
BUSU*ESS  BASIC 

UNICOMP 
COMP  16  or 

LQ."P   AO  BAolL 

VARIAN 
620  or  V73 

SCRATCH 

ERASE  name 

SIHTTR 

REKEY 

^lALLESr  *  ** 

lE-99 

8.758E-47 

in-99 

lE-99 

lE-99 

l,67r>57 

lE-99 

STOP 

STOP 

STOP 

STOI> 

STOP 

STOP 

STOP 

STRIKG  QUCTTES  ** 

ii 

I* 

II 

II 

STRINGS  -  ^ 

STRIW  VARIABLE  WME  ** 

letter  S 

Hum  name  S 

SUB 

SUBHND 

TIME 

"""""""" 

IWtESS 

USER  DEFINE)  FUNCTIOK 

FN  Ifttter 

FN  letter 

FH  letter 

FH  letter 

F.S  letter 

FN  letter 

9I> 


**Comands  and  ele«ents  that  caji  be  osed. 


im  S3 

MOO  6 

a\sic 

TIME  SHARirr 
BASIC 

ccw-suyiE 

BASIC 

CCM-S1!,^RE 
KEVa^SIC 

WESTINaKWSB 
BASIC  IT 

h'ESnSGHOUSE 
BASIC  III 

GENERAL 
AUTOttTTON 
AIVAVCH) 
HASTC-16 

see  RESET 

RESTORE  *  exp 

SCRATCH 

9t,RATut  exp 

SnPTR 

SWiLEST 

15-99 

S,78960E-76 

sn-76 

SE-76 

2,7lE-20 

2,715-20 

2. 715-20 

STOP** 

STOP  cofTtcnt 
or  STOP 

STOP 

STOP 

STOP 

STOP 

STOP 

STOP 

S^RI^G  QLOTES** 

1 

I* 

"  or  ' 

tt 

STRi:OGS 

letter,  St  4*  or 
1  fallowed  by  S 

letter  J 

ntm  nfiiK  J 

ntm  nnnE  £ 

letter  S 

letter  S 

QIC 

SllBEND 

"  ~ "  -  -  -  -  - 

TIME 

TIME 

- "  ~ " " "  ~ " " 

"""""""" 

UNLESS 

UNTIL  exp 



USER  DEFINED  FUNCTION 

^A^2S** 

FN  letter, 
or  # 

FN  letter 

FN  letter 

. 

EN  icmr 

1 

1 

letter 

Fit  letter 

letter 

'CbrmiKis  and  elements  that  can  bo  iised. 
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UNIVAC  1100 

ICNEVHELL  1640 
XBA5IC 

HnKElrtflBLL 

TlA  <md 

716  BASIC 

HGHEViCLL  400 
XBASTC 

HONEn«Ea  600 

KP2000E 

UNIWC  1100 
ISflV  OF  MlRVf jwn 

RELEASE  V  1,3 

RESTORE  1  tvp 

RESTORE  f  iwm 

RESTORE  *  nua 
RESTORE:  i)u:!i 

scmai 

SCRATOI 

inane: 

SCf^TCH  f  imm 

SOlATOi  ' 
SCRiMQI:  nisn 

_  _  _  _  _  _ 

SETPTR 

SET:  mDTi  10  exp 

a^WUEST  *** 

IE- 39 

lE-38 

lE-38 

S,7S96B-76 

1E-3B 

IE- 38 

1E-3& 

STOP** 

STOP 

STOP 

STOP 

STOP 

STOP 

STOP 

STOP 

It 

II 

II 

It 

» 

II 

STRlrCi  VARlflSLB  *WMc"" 

setter  4 

nuR  nunc  ^ 

letter  * 

letter  * 

SUB 

TIME 

UNLESS 

UNTIL  condition 

USER  DEFINED  HWOflOK  NAMES** 

FN  letter 

FN  letter 

TS  letter 

FN  letter 

FN  letter 

FN  letter 

Fa  letter 

'CoRcn^ds  and  elerents  that  can  be  used.' 


77 


>(lCn)QATA 

i^i;ic 

Q-EATA 

MANG  3300 

GENERAL 
ELBOTtlC 

mty.  I 

WMG  2200 

REHIND 

RESTCRB  I  exp 
RESTO^^I■;  exp 

SCRATCH 

FI13tOO  I  exp, 
option 

— 

SCRATOI  f  cxp 
SCRATCH:  Qxp 

SETPTH 

SET;  esp,  var 

lE-99 

lE-77 

lE-05 

4.3ia09E-78 

lE-100 

STOP** 

STOP 

STOP 

STOP 

STOP 

STOP 

STOP 

STOiP  "comnenf* 
STOP  tiigit 

STRING  QDDTES** 

■t 

If 

STRINGS 

STRING  VARIABLE  >WME»* 

letter  $  or 
letter  digit  S 

letter  S 

letter  t 

nun  name  S 

SUB 

DlcfPS'  mm 

(VTir  1,        war  lO 

DEFW  (string) 

SUBEM) 

TIME 



UNLESS 

iwriL 

USER  D£PrriED  FtftCnOM 
XV^fES** 

FK  letter 

FN  letter 

FX  letter 

FN  letter 

letter 

letter 

"Conmncb  and  elements  that  can  be  used. 


BASIC  2.0 
CDC  6600 
SCOPE 

IM 

CPS 
UNI\*  OF  lOWl 

GEMWX  II 
GE>WKK  III 

HP2000S 

(1P2000C 

WHILE 

NRITE  FUE 
(itaae)  list 



WIIE  *  eicp:  list 

kKm  FILE 
(exp3  list 
or 

VKFre  FILE  (exp^ 
exp}  list 

WRITE  *  expp  list 
t^TtlTE:  exp*  list 

see  PRIta  ' 

SM  PRim'  1 

WillE  1  USING 

USING  str,  list 

miB  C  -  ) 

WITH  1 

5««  PRIffT  f 

SM  PRINT  1 

s 
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ITF 

LEASOO 

10" 

PDF  11 

Ur^IVAC  1100 
UBASrc  VERSION 
htfmiO  STATE  CLG 

MJLTICCMP 
OR 

USrV  MASS 
EASICX 

MtaLE 

MIILE  cordition 

MUTE  f 

ivr  *iuDifr\ 

list 

WRITE  f  list 
MUTE;  N(  list 

see  PRINT 

WRITE  OK  nvDi: 
list 

see  WIITE  (  ,  ) 

Icey,  list 

WRITE  ■  UolNG 

MUTE  (  ,  ) 

see  PRIKT 

WRITE  Cm,  n) 
list 

lOS 


r— 


ERIC 


SI 


IBM 

CALL/360 -OS 

PDP  $/E 

HONEYWELL 
200 

CDC  $000 
KRONOS 
liASIC  2.0 

NCR 

CE^^TURY 

200 

UCSD* 
BASIC 
B6700 

HP2000F 

WHILE 

PUT  exp: 

KRITE  f  expp 
v«r  t  p  *  *  * 

KRITE  FILE 
(n3me)expt  ^ . . . 

see  PRINT  1 

see  PRINT  » 



'  USIKG 

see  WRITE  1 

see  WRITE  1 

see  WRITE  ' 

see  PRINT  « 

sec  PRIKT  1 

"UNIVERSITY  OF  CALIFORNIA,  SAN  DIEGO,  BURROUGHS  56700 


ERIC 


32. 


NCR 
CENTURY  100 
BASIC  1 

BURROUGHS 
BSSOO 
BASIC 

BURROUGHS 
BZSOO 
3.\SIC 

Bl/RROUGHS 
fi3S00 
BASIC 

BASIC  FOUR 
BUSINESS  BASIC 

1 

UHID)KP 
COMP  16  or 
COMP  IS  BAsrc 

VARIAN 
620  or  V7  3 
BASIC 

WHILE 

WRIIE  # 

see  PRINT  # 

WRITE  (mtttf 
IKD-exp,ERR- 
nunif  BKD^  num) 
list 

WRITE  t  USING 

______ 

________ 

KRITE  USING  ttm 
(in>n,lM>exp,BRI^ 

see  PRINT  ' 

s«  :flITE  1 

lOD 


rEM  S3 
WOO  6 
BASIC 

TIME  S!!ARI>« 

Br^^SlC 

IVkSlC 

\TARr>ISTC 

Br\sic  n 

BASIC  III 

GENERAL 
BASIC- 10 

V.'HILE  ex? 

RJT  nttpi^p  var  1, 
var  Z,  ... 

var  1,  var 

fe^  PRI^T  f 

sec  rRL\T  nw 

lAlTB  »  USING 

WIIB  C  •  ) 

sec  MtlTE  i 

see  MtllB  * 

S«  PRINT  # 

see  PRi:/r  FRCM 

10£ 


ERIC 


Si 


UNIVAC  lliK) 
UBAStC 

HONEYWELL  1640 
XBAStC 

HONEYWEtX 
316,  516,  and 
716  BASrC 

;    HQKEVWHX  400 
3CBASIC 

HONEyWELL  600 
BASIC 

HP2000E 

UKIVAC  1100 
UNIV  OF  MWYLAKD 
RELEASE  V  1.3 

WHILE 

WILE  condition 

tRiTE  1 

IrtinE  OK  exp; 
list 

WRHE  '  exp, 
list 

list 

WRHE  ^  ntm, 
li-<t 

WIITE;  mm*  list 

hUlTE  f  USING 

see  WIITE  ' 

KRITE  if  expp 
niTOp  lift 

WRITE  C  ,  ) 

see  MftlTE  » 

sce  hTUTE  f 

see  KRTTE  * 

lOF 


ERIC 


MtCRODATA 
BASIC 

BASIC- 1 

ELECTRIC 
M^RK  I 

>et^  Z2D0 

miiM 

VtRlTB  9 

see  PRINT  ' 

FlLEhHITE 
f  nun,  list 

MllTE  f  cxp, 
var  i,        var  n 
VHITE  -  expj 
var  1,  ^     var  n 

DATASWE 

KRITE  t  USING 

MIITE  C  ,  ) 

sec  Rllffr  ^ 

sec  WRITE  t 

see  \MTB  1 

see  KRITE  * 

lOG 


00 


ERIC 


ERIC 


The  following  ^  cables  are  n  list  of 

ABS:  =  Absolute  value 
ACS:  =  Ascii 
ASN;  =  Arcsin 
ATN;  -  Arctangent 

BOOL:  =  Returns  true  value  of  relation 

CLK:  =  Time  of  day 

COL:  =  Next  print  position 

COS;  =  Cosine 

COT:  "  Cotangent 

CSC;  =  Cosecant 

CSF:  "  Returned  statuscode 

DAT:  =  Date 

DEG:  =  Degrees  from  radians 
DET:  -  Detemdnent 

DIG;  -  Digital  part  from  scientific  notation 

DlV;  =  Integer  division 

EOF;  =  End  of  file 

EFT:  -  Exponent  parr, 

EXP:  ^  Exponentiation 

FIX:  Truncation 

FLD:  -  Selects  bits 

FRP:  *  Fractional  part 

GET;  -  Field  data  equivalent 

HCS:  =  Hyperbolic  cosine 

HTN:  »  Hyperbolic  tangent 

INP;  *  Integer  part 

INS:  »  Converts  to  binary  Integer 

INT:  *  Largest  integer 

KEY:  =  Next  available  position  of  file 


86 

tho  BASIC  built-in  functions  vjhere: 

LIN:  =  Skips  lines 

LOC:  =  Location  of  file  pointer 

LOF:  =  Length  of  file 

LOG;  =  Natural  logarithms 

LOGIO:  -  Common  logarithms 

LTW:  =  Logarithm  base  2 

MAR:  =  Margin  for  file 

MOD:  =  X  Y*IKT(X/Y) 

MXL:  «  Maximum  length  of  string 

NUM;  »  Number  of  data  input 

PI:  =  3tl415927 

FIX:  =  TT  times  argument 

PCS;  =  Location  of  string  on 
RAD;  «  Radians  ^ 
REKEY;  "  Change  position  number  of  record  in  file 


END: 

Random  number 

RUN: 

Elapsed  time 

SEC: 

Secant 

SGN; 

cc 

Algebraic  sign 

SIN: 

Sine 

SPA: 

Skips  spaces 

SPC: 

Outputs  a  number  of  spaces 

SQR: 

Square  root 

TAB: 

Tabulation 

TAN: 

Tangent 

TIM; 

Elapsed  time 

TIS: 

Time  of  day  in  milliseconds 

TYP: 

Type  of  file 

XPT: 

Exponent  part 

b7 


AfiS 

ACS 

ASN 

COL 

COS 

COT 

CSC 

CSF 

DAT 

DIG 

DIV 

EXP 

FIX 

FU) 

FRP 

GET 

HCS 

HSN 

INP 

INS 

BASIC  2*0 
€OZ  6600  SCOPE 

X 

X 

X 

X 

X 

X 

im  CPS 

X 

X 

X 

X 

X 

X 

X 

¥ 

X 

X 

X 

¥ 

¥ 

¥ 

X 

X 

X 

uc  MARK  il  Q  III 

X 

¥ 

X 

X 

X 

X 

ririUUUs 

X 

X 

X 

X 

V 

\ 

X 

X 

X 

X 

I&I  ITF 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

/V 

LEASCO 

X 

X 

X 

X 

X 

V 

\ 

¥ 

X 

X 

X 

X 

X 

JrUr  11 

V 

\ 

¥ 

¥ 

X 

X 

X 

Ui«t  ^        1  Ivv  vcn 

:;W!CATO  STATE  CLG 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MUITICOMP  OR 

X 

X 

X 

X 

X 

X 

X 

XEROX 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

la^  CA1X/360-OS 

X 

X 

X 

X 

X 

t 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

HCKEYKELL  200 

X 

X 

X 

X 

X 

X 

CDC  6000 
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X 

X 

X 

X 

X 
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¥ 

X 

1 

X 

X 

X 

X 
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X 

X 

¥ 

¥ 

¥ 
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X 

X 

X 
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X 
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X 

X 
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X 
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X 

1 
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X 
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ABS 

ACS  ' 

ASN 

ATN 

BOOL 

COL 

cfis 

cor 

CSC 

CSF 

DAT 

DIG 

DIV 

EOF 

EPT 

EXP 

FID 

GET 

ICS 

WIN 

I>S 

INT 

KEY 

BURROUGHS- ^SQO 

ATAN 

X 

X 

X 

X 

X 

FPT 

X 

X 

X 

X 

VARlAN-fZDp  V73 

X 

X 

X 

X 

X 

IBM  SJ  MDD  6 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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C£  2jS  time  sharing 

X 

X 

X 

X 

X 

X 
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X 

X 

X 

X 

X 
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X 
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ATAN 

X 

X 

X 

COSif 

TANH 

X 
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X 

X 

X 
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X 

X 

X 

X 

X 

G&'ERAL  jUJTOWLTlCN 
BASIC- 16  ADvAXCQl 

X 

X 

X 

X 

IINT\'AC  1100  UHASIC 
VERSION  3.2 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

HQNlffKELL  1640 
XEASIC 

X 

X 

X 

X 

X 

fljNEYWELL  316p  516^  716 

X 

X 

X 

X 

X 

HOSEir*"TLL  400 

X 

X 

X 

X 



X 

HD^fHVKELL  600 
E^StC 

X 

X 

X 

X 

X 

X 

X 

IIP^OOOE 

X 

X 

X 

X 

X 

twn'Ac  1100 

UNrV  OF  MARYLAND 

X 

X 

X 

X 

X 

X 

X 

X 

MICRDDATA 

X 

X 

X 

X 

X 

Q-DATA  BASIC- 1 

X 

X 

X 

X 

X 

¥ 

¥ 

¥ 

TVU 

irvi 

V 
A 

KVOG  3500 

X 

X 

X 

X 

X 

GEMARK  I 

X 

X 

X 

X 

X 

X 

tasx;  2^00 

X 

ARCOOS 

ARCSIN 

X 

X 

X 

X 

LIN 

UK 

WG 

LOGIO 

LTK 

MAR 

HDD 

MXL 

pi' 

PDC 

POS 

RADj  R£KDr 

RND 

RUM 

Sir 

SGS 

SLV 

SPA 

SPC 

SQR 

TAB 

wt 

TIM 

TIS 

XPT 

BASIC  2.0 

X 

X 

X 

X 

X 

X 

X 

uNiv  OF  mx 

X 

ALGT 

X 

X 

X 

X 

DARTMOUTH 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

a^TA  GIJaiAL 

X 

X 

X 

X 

X 

X 

GE^tARK  II  a  III 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

ttPZODOC 

X 

X 

X 

X 

X 

X 

X 

X 

I  EM  ITF 

X 

X 

Y 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

POP  10 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

?DP  11 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

IS^IVK  IIDO  VHR  2.0 
HAXKATO  STATE  CLG 

X 

LOT 

X 

X 

X 

X 

X 

X 

X 

X 

ns 

X 

USIV  MASS  BASia 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LGT 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LGT 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

KRQN'OS  BASIC  2.0 

X 

X 

X 

X 

X 

X 

X 

X 

CCM 

X 

X 

X 

X 

X 

X 

tCSD  BASIC  B6700 

X 

X 

X 

X 

X 

X 

X 

HP2CD0F 

X 

X 

X 

X 

X 

X 

X 

NCR  CENTURV  100 

X 

X 

X 

X 

X 

X 

ei;rrcug.^-bs5oo 

X 

X 

X 

X 

X 

X 

X 

X 

X 

HJRPOUaiS-BZSOO 

X 

X 

X 

X 

X 

X 

X 

00 


FRir 


90 


LIN 

LX 

LOG 

LOGIO 

LTW 

MAR 

MOO 

wet 

PI 

PIX 

POS 

RAD 

REKEf 

RUN 

SEC 

srN 

SPA 

SPC 

SQR 

TAB 

TAN* 

TIM 

TIS 

TYP 

XPI 

BURROUGHS- B3S00 

i  ■ 

X 

X 

X 

X 

X 

X 

X 

BUSINESS  BASIC 

¥ 

A 

X 

X 

X 

VA?iAN-6^J,  V73 

X 

X 

X 

X 

X 

X 

X 

lEM  S3  >(0D  6 

X 

LGT 

X 

SPl 

X 

X 

X 

X 

X 

X 
X 

X 

GE  £55  TBlE  SKWING 

X 

X 

X 

X 

X 

X 

Cai-SltAA£  BASIC 

X 

LGT 

X 

X 

X 

X 

X 

X 

X 

COM-SltlU^  NDhBASIC 

X 

X 

X 

X 

X 

X 

X 

X 

X 

WESTLSSQIOOSE  BASIC  11 

X 

X 

X 

X 

KESrtNQJOUSH  &\5IC  III 

X 

X 

X 

X 

X 

X 

X 

X 

CD;ERM  AlfTOttTlCH 
aA$lC-I6  AEA'ANCED 

)^ 

)^ 

V 

\ 

X 

X 

X 

X 

X 

I^IVAC  1100  UEASIC 
VERSrCW  3.2 

X 

LGT 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

XEASIC 

X 

X 

X 

X 

X 

X 

X 

X 

H0^'EWELL316,  516,  716 

X 

X 

X 

X 

X 

X 

X 

XBAStC 

X 

OC 

X 

X 

X 

X 

X 

X 

X 

HOSEYlii^LL  600 
BASIC 

X 

CLG 

X 

X 

X 

X 

X* 

X 

X 

X 

IITZOOOE 

X 

X 

X 

X 

X 

X 

X 

X 

X 

UNEVAC  1100 
UNrV  OF  MARYIAND 

V 

\ 

V 

\ 

V 

\ 

X 

V 

\ 

X 

X 

X 

X 
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X 

X 

X 

X 

X 

X 

X 
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X 

X 

X 

X 

X 

X 
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X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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X 

X 

X 

X 

X 

X 

X 
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X 

X 

X 

X 

X 

X 

X 

X 
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X 

m 

X 

X 

X 

X 

X 

X 

The  following  table  refers  to  matrix  operations  and  built-in  functions  wher 


CON:  =  Matrix  of  a!  -  ones 
IDN:  =  Identity  matrix 
INV:  =  Inverse 

NUL$;  =  Matrix  of  null  strings 
TRN:  =  Transpose 
2KR:  "  2ero  matrix 
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A-B 

KaA 
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im 

ZER 
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The  next  table  shows  various  string  built-in  functions*    These  functions 
are  sometimes  quite  complicated  and  their  descriptions  should  be 
referenced  In  the  appropriate  manual. 
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cwiicxj 

ASClXCAi) 

- 

RiGHrcAJ,ro 

cues 

DATS 

PER 

POS 

SIR 

sec 

TYP 

cm 

1 

Exit 

CPYi 

ADDS 

SPACES 

INSIKf 

BASIC  2.0 

CDC  smsoopB 

jm  CPS 

UKrV  OF  ICMK 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

tmA  GENERAL 

X 

<3EMARK  lU  Hi 
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X 

X 

X 
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X 

tIP2000B 

X 

IIP2000C 

X 
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FDP  10 

X 

A3C 
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Ricarj 
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X 

X 
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X 

X 

X 

X 

X 
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X 
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fltSKATO  STATE  CLG 
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X 

X 

X 

X 

X 

X 

MtfLTlOCMP  OR 
U^R'  MASS  BASia 

XEROX 

X 

X 
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X 

X 
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X 
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X 
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X 
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X 

X 
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X 
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X 
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X 

UNIVAC  1100 
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X 
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X 

X 

X 
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X 
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X 

X 
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COM-SHARE  NEWBASIC  extensions 

1»  LET  VAR  =  ZERO — zeros  ctll  variables » 

2*  Statements  may  contain  up  to  256  characters* 

3*  Complex  variables 

4,  Data  type  declares—INTEGER,  DOUBLE  INTEGER,  COMPLEX,  REAL,  DOUBLE  REAL,  STRING,  TEXT 

5*  Very  much  less  than  « 

6,  Very  much  greater  than  » 

7»  Binary  operators — BAN  conjunction,  BOR  disjunction,  BEX  exclusive  or 

8»  Loglal  operation — BUT 

9-  Allows  mixed  data  types  and  converts* 

10»  Comments  may  be  added  after  any  statement, 

11*  Suffix  modifiers  may  be  added  after  any  non  declarative  statement. 

12*  Keywords  may  be  abbreviated* 

13-  LET  var  "  exp  1  -  exp  2 

14,  NORMAL  MODE  IS 

15*  DIM  var  (exp:  exp)  as  in  ALGOL 

16*  LINK  saves  variables,  LOAD  does  not*  Ob 

17*  May  LINK  or  LOAD  BINARY  ^ 

18*  APPEND  in  execute  mode 

19*  FOR  var  =  exp  1,  exp  2,  .** 
FOR  ***  UNTIL  or  WHILE 

20,  May  use  brackets  [  ] 

21,  ON  ESCAPE  GOTO  line  # 

22,  ERASE  exp  FROM  exp  TO  exp  deletes  material  on  random  file 

23,  Formatted  Input— INPUT  IN  FORM,  INPUT  IN  FORM  FROM 
24*  Sentlng  BASE 

25*  Suffix  modifiers  FOR,  IF,  UNLESS,  UNTIL,  WHILE 
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26*    String  functions;    lEQIV — searches  for  substrings 

CTI — character  to  integer 
ITC — integer  to  character 
SPACE — returns  spaces 
LITRIM — removes  leading  blanks 
TRIM — removes  trailing  blanks 
INDEX — returns  position  of  substring 
27*    Functions :    DIF — positive  difference 

J  FLOAT — floating  point  of  integer 

SHGL — single  precision  from  double 
LSH~left  shift 
RSH^right  shift 

7HAG — imaginary  part  of  complex  number 

REAL~real  part  of  complex  number  O 
COMPLX — complex  number 
CONJG — conjugate 
WAIT — halts  for  time 

PASS — number  of  times  statement  is  executed 
REPASS — resets  PASS 
DATE — 12  character  date 
TEL — tells  if  terminal  buffer  empty 
SIZE — length  of  file  in  words 
28*    Catalyst  functions 


ERIC 
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Extensions  of  HP3000 

BASIC 

1. 

Continuation  of  statements  t>y  placing  &  as 

last  character 

2. 

Double  precision  variables 

3. 

Complex  variables 

4, 

Integer  variables 

5, 

TYPE  statements  INTEGER  COMPLEX  LONG  REAL 

6. 

Rediraensioning  by  REDIM 

7. 

IF  -  DO  and  DOEND  pairs 

8. 

For  loops  in  READ  statements 

9, 

May  save  extra  INPUT'S  in  a  buffer  and  BUF 

function 

10. 

Complex  functions  CEI,  CPX,  REA,  IMG,  CNJ 

11. 

String  functions  WRD,  UPS,  DEB 

12- 

Matrix  functions  ROW,  COL  * 

13. 

Functions  UND,  CPU,  REC 

14. 

May  define  type  functions 

15. 

Call  external  procedure  in  other  libraries 

by  EXT 

36. 

CPDATE  allows  file  to  be  modified. 

ACT     TECHNICAL     BULLETIN     NO,  12 
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THE  CADA  MONITOR 


David  E*  Christ 
The  University  of  Iowa 

Several  elements  go  Into  a  Bayeslan  statistical  analysis*    Some  are 
skilled  tasks  requiring  the  expertise  of  a  professional  and  others  are 
purely  mechanical*    The  former  Include  such  tasks  as  choice  of  model, 
specification  of  the  prior,  and  Interpretation  of  the  posterior  dletrl- 
butlon;  whereas  the  latter  Include  such  things  as  the  arithmetic  necessary 
to  take  statements  about  the  prior  and  combine  them  with  the  data  to 
produce  the  posterior  distribution  and  to  produce  probability  statements 
about  parameters  using  the  posterior  distribution*    Unfortunately,  It  Is 
all  too  often  the  case  that  the  arithmetic  gets  In  the  way  of  the  pro- 
fessional's decision-making  task  by  breaking  concentration  and  line  of 
thought;  and  at  times  the  sheer  bulk  of  computation  precludes  the  use  of 
advanced  techniques  by  the  unaided  researcher*    For  these  and  other  reasons, 
a  system  of  Computer^Aasisted  Data  Analysis  (Novlck,  1971)  was  developed 
at  The  University  of  Iowa*    Farther  Investigation  Into  available  computer 
technology  coupled  with  expansion  of  the  theoretical  base  on  which  the 
original  system  rested  has  resulted  in  the  refinement  and  expansion  of 
the  available  programs  and  the  construction  of  a  monitor  to  facilitate 
their  use* 

Since  CADA  (Computer-Assisted  Data  Analysis)  was  meant  as  a  research 
tool  for  general  application,  a  search  was  made  to  find  the  most  effective 
means  of  facilitating  wide  distribution  of  the  njonitor  for  use  on  many 
computing  iyitasa*    Dua  to  limitations  in  time,  manpower,  and  money, 
reprogramlDf  on  a  •yst«m<"by*system  basis  was  rejected  as  a  viable 
method  of  implementing  CADA*    Since  no  entirely  transportable  languagtj 


for  all  interactive  systems  existed^  it  was  decided  to  pursue  a  strategy 
which  would  permit  interdialict  translation  rather  than  actual  repro- 
gramming.    Examination  of  available  hardware  and  software  pointed 
toward  the  BASIC  progranuaing  language  &s  the  only  possibility  for 
translatability  across  several  manufacturers.    A  study  was  then  made 
by  Isaacs  <19?2)  which  showed  that  programs  written  in  one  dialect  of 
BASIC  could  easily  be  translated  into  that  of  many  other  manufacturers' 
dialects  provided  certain  specified  constraints  on  the  initial  programs 
were  observed.    The  first  BASIC  version  of  CADA  was  then  written  by 
Isaacs  and  Christ  in  the  BASICX  dialect  for  the  CDC  3600  at  The  University 
of  Massachusetts.    This  was  then  easily  and  quickly  translated  into 
versions  for  the  Hewlett-Packard  2000C  and  the  Digital  Equipment 
Corporation  PDP-11>  thus  validating  the  assertions  made  by  Isaacs, 

The  detailed  outline  of  the  current  monitor         developed  based  on 
considerations  falling  in  three  basic  areas — user  interaction^  systems 
constraints^  and  programming  considerations.    The  user  interaction  is 
by  far  the  most  Important  consideration*    Although  the  user  may  be 
highly  skilled  in  his  own  subject  area>  he  may  be  quite  unsophisticated 
in  terms  of  computer  skills.    The  first  design  rule  was  then  that  the 
user  be  required  to  have  no  programming  skills.    He  need  know  only  three 
sysstem-related  commands;     (1)    how  to  sign  on  the  system;  (2)    how  to 
start  the  monitor  running;  and  <3)    how  to  sign  off  the  system. 

The  second  design  rule  was  that  the  monitor  be  sslf-documenting 
in  terms  of  options  available.    The  monitor  should  be  modifiable  to 
include  new  models »  new  techniques »  and  Improvements  to  current  programs 
without  the  user  having  to  wonder  whether  he  has  the  latest  "newsletter" 
or  update  she^t. 
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The  third  design  rule  was  that  the  user  should  not  be  left 
"hanging".    If  a  numerical  Integration  falls  to  converge,  an  error 
message  followed  by  the  stopping  of  the  program  Is  not  enough.  Control 
must  branch  to  a  point  where  the  unsophisticated  user  can  proceed  on  the 
Information  available  to  him*    Furthermore,  whenever  possible^  Input  from 
the  u&er  must  be  checked  for  validity  to  avoid  system  errors  such  as 
division  by  zero,  taking  the  root  of  a  negative  number,  etc. 

The  constraints  of  any  language  Implementation  limit  what  can  be 
prograimned  In  that  language.    When  programnlng  for  trans latablllty  across 
several  systems,  the  constraints  become  somewhat  more  demanding  and  at 
times  preclude  the  use  of  features  that  may  be  present  on  one  system 
only,  or  that  differ  radically  from  one  system  to  the  next*    This,  with 
the  three  design  rules  mentioned  above^  has  governed  most  of  the  design 
of  the  monitor  and  the  programs  * 

While  the  monitor  Is  currently  available  for  operation  on  only 
three  systems^  an  attempt  has  been  made  to  minimize  the  dependence  on 
features  not  available  In  BASIC  dialects  for  other  computers.    The  two 
features  used  which  might  be  the  most  limiting  are  chaining  and  formatted 
print  statements*    However^  the  systems  In  which  we  are  most  Interested 
have  these  features  available*    The  formatted  print  statements  were  used 
to  present  the  output  and  textual  material  In  a  visually  pleasing  way* 
This  Is  not  necessary,  per  se,  but  Is  desirable  to  facilitate  the  man- 
machine  Interaction  since  the  Intended  user  Is  not  presumed  to  be  a 
computer  expert*    The  formatted  print  statements  do  have  analogs  In  the 
other  dialects  we  propose  to  use;  however^  they  will  be  the  ones  needing 
the  most  change  from  machine  to  machine* 
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Chaining,  which  Is  nece&&ary  In  some  larger  machines  and  most 
smaller  machines »  Is  much  more  central       the  logical  design  of  the 
sy&tem*    The  first  consideration  wa&  that  the  u&er  need  only  know 
how  to  sign  on  the  sy&tem  and  would  not  need  to  know  the  names  of  the 
Individual  routines*    This  implies  either  a  main  routine-subroutine 
sy&ten  or  a  monitor  program  which  cau&es  the  loading  of  the  proper  program* 
The  latter  Is  the  sy&tem  used  by  us^  dictated  by  the  design  of  most 
BASIC  systems*    The  main  rout In e*subrou tine  system  has  the  advantage  of 
ease  of  parameter  passing*    However^  the  number  of  parameters  to  be  passed 
In  our  system  Is  few  ^nd  the  values  are  values  knovn  to  the  u&er»  u&ually 
understood  by  hlm^  and  normally  recorded^  to  be  used  In  any  published 
record  of  the  analysis;  thus^  It  Is  reasonable  to  ask  the  user  to  reenter 
th*^  parameters  when  nece&sary*    This  also  allows  the  u&er  to  ea&lly  do 
an  analysis  In  steps  at  different  tlme&.    The  chaining  a&  used  here  has 
the  advantage  of  having  In  core  only  the  program  in  use  and  thus  reducing 
system  overhead*    A  second  consideration  for  the  system  Is  that  it  should 
be  ^ixpandable  with  little  effort  on  the  part  of  the  programmer  and  with  no 
operational  change  vl&lble  to  the  user*    The  monitor  system  used  here 
permits  thl&*    The  only  change  seen  by  the  u&er  Is  that  he  Is  given  the 
choice  of  choosing  among  a  larger  set  of  routines  and  technique&.  The 
programmer  need  add  only  about  three  llne&  of  coding  to  the  monitor  to 
make  a  new  routine  available  to  the  user.    A  third  consideration  Is  that 
the  user  should  never  be  left  dangling  after  he  make&  an  error.    In  the 
CADA  monitor,  when  a  program  fails^  the  system  chains  to  &  routine  In 
which  the  user  la  told  to  save  the  output  for  use  by  the  person  maintaining 
the  system  and  is  then  returned  to  the  monitor  to  continue  the  &es&lon  If 
he  so  wishes*    All  user  Input  1&  screened  for  validity*    Since  string 
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handling  capability  Is  not  highly  developed  In  all  BASIC  dialects  and 
handling  a  finite  set  of  responses  can  be  done  by  much  simpler  coding, 
user  responses  to  questions  within  the  program  segments  have  been  forced  to 
numeric  form* 

Programing  ease  was  also  considered.    A  modular  method  was  used 
i'l  building  the  routines  themselves  *    Many  routines  were  common  across 
programs  (e.g^^  Integrating  a  beta  distribution,  calculating  an  Inverse 
chl  highest  density  region)  and  were  assigned  specific  line  numbers  above 
5000*    These  routines  were  coded  only  once  and  after  being  debugged  were 
usable  without  further  effort  on  the  part  of  the  programmer*    The  programmer 
then  referenced  these  routines  by  GOSUB  statements  to  predetermined  line 
numbers  with  no  need  to  worry  about  where  to  put  them*    Unique  portions 
of  programs  were  then  programied  with  line  numbers  below  2000*    As  noted 
above^  the  monitor  system  used  enables  new  programs  to  be  added  with  little 
programming  efiort* 

The  accompanying  appendices  show  a  sample  of  the  monitor  output, 
give  a  listing  of  the  current  package  contents,  and  outline  the  chaining 
sequence « 
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APPENDIX  I 
Monitor  Output 


•^UtJ  C3CADA 

COMPUTER  ASSISTED  DATA  ANALYSIS 

IF  YOU  WISH  AW  EX'^LAWAT  ION  TYPE  1/  ELSE  TY^JE  S 
?  I 


THIS  "ACKET  OF  PIOGHAMS  "^-^oyiDES  A  GROUWDIWG   IN  THE 
FU-:^DAMEMTALS  OF  BA^ES  IAN  HETHOnS  OF  STATISTICAL  INFEnENCE, 
THESE  ROUTINES  ARE  DESIG^JED  TO  GUIDE  THE  nESEA-'.CliER  WHO  1:AS 
ONLY  A  tllHIMAL  AClUA  IWTENCE  VITH  3AYES  lAM  METHOnS.  STEP-3Y- 
STE^  THROUGH  A  COMi^LETE  3AYE5  IAN  ANALYSIS.     A  LIST  OF  THE 
ROUTIEJES  FOLLOWS: 

1.  -^RlOn  BETA-DINOMIAL  MODEL 

2.  POSTERIOR  BETA-3IN0^!IAL  MODEL 

3.  "3RI0R  TWO  PARAMETER  NORMAL- -MAIG IJJAL  DIST  FO:^  STAJJDARD  Di:;V 
it,  PRIOR  TWO  "=A^AMETE?l  NORMAL- -COWD IT  lO.^AL  DIST  FOR  MEAN 

5,  POSTERIOR  TWO  PARAMETER  NORMAL 

6,  PRIOR  M-GROU"  PRO'^ORTIOHS 

7,  POSTERIOR  M-GROUP  ^RO^^ORTIOWS 
e.  EVALUATE  STUDENT -D I STR IQUT ION 
9.  EVALUATE  BETA-D ISTR I3UT ION 

l^,  EVALUATE   INVERSE  CH I -D ISTR IDUT ION 

11.  EVALUATE  NORMAL  DISTRIBUTION 

in,  CALCULATE  MEANS .  STANDARD  DEVw   SUMS  Or  SQUARES 

IF  YOU  UANT  TO  RUN  ONE  OF  THE  A30VE  ROUT  INES.  TY^E   ITS  MUM.IER 
OTHERWISE  TY-^E  A  ZERO. 
?  I 
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APPENDIX  II 


Package  Content& 

I.    Supervl&ory  Routines 
A*    CADA  -  Monitor 

B,    ERROR  -  Gives  Instructions  when  a  program  falls 

II.    BETA  -  Binomial  Model  Routines 

A.  PRIORB  -  Assists  In  fitting  prior  knowledge  to  the  beta  class 

B.  POSTS  -  Combines  a  beta  class  prior  with  binary  data  to  give 
a  beta  posterior 

III*    Two  Parameter  tformal  Model 

A*    PRIORS  -  Fits  prior  knowledge  (marginal)  on  the  standard 

deviation  to  an  Inverse  chl  distribution 
B,    PRIORM  -  Fits  prior  knowledge  (conditional)  on  the  miian  to  a 

normal  distribution 
C*    POSTN  -  Combines  the  Inverse  chl  and  normal  priors  with  normal 

data  to  give  posterior  distribution 

IVt    m-Group  Proportions 

A«    PRIORP  -  Evaluates  exchangeable  prior  Information  on  any  of  a 

set  of  proportions  for  use  In  an  m^-group  proportion  routine 
Bt    PROPOR  -  Solves  the  Llndley  equations  for  a  set  of  binary  data 

V,    Evaluation  Routines 

A*    TDIST  -  Evaluates  the  probability  Integral  of  a  nonstandard 

student  t-dlstrlbutlon 
B»    BDIST  *•  Evaluates  the  probability  Integral  of  a  beta  distribution 
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C,  ICDIST  -  Evaluates  the  probability  integral  of  a  nonstandard 
Inverse  chl  distribution 

D,  NDIST  -  Evaluates  the  probability  integral  of  a  nonstandard 
nonnal  distribution 

VI.    Service  routine  STAT  calculates  the  mean»  standard  deviation^  and 
Sum  of  squared  deviations  from  the  mean  for  a  set  of  data 
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APPENDIX  III 


Chaining  Sequence 


tXXtiX  IDA/  b«  C«Il«d 

froci  ^tty  ftcdul«  upon 
d«t«ctloD  of  «ti' 
«rror  or  «bnonul 

COTtdltlOft. 

ERF^K 

«^ 

— J 

 J 

TDIST 


IC&IST 


PROFOK 


Note;    Any  program  can  chain  to  error  upon  detection  of  an  abnormal  condition 
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